1
$\begingroup$

In a neural network, a bias is a constant term that is added to the weighted input in a neuron/unit: output = activation_function( input1*weight1 + ... + inputn*weightn + bias)

I can see that the bias in a sigmoid activation function adds the ability to control the threshold of the activation. But when I started learning about neural nets we didn't use any bias, just the weighted inputs. I've also been told that sigmoid in particular can do without a bias, but this is not intuitively true for me at all.

So is it true that biases are redundant in sigmoid neural nets? How can neural nets learn to approximate any continuous function if they do not have a bias input?

$\endgroup$
3
$\begingroup$

No, you don't need a bias. You can have a "dummy" input (input(n+1) in your formualtion) which is always set to 1. Then the bias term is absorbed into the weights.

$\endgroup$
  • $\begingroup$ That's still a bias though. You're just shifting the work on the user instead of the internal logic of the system. $\endgroup$ – mtanti May 15 '16 at 13:38
  • 6
    $\begingroup$ In that case, it seems that without a bias and using only "unshifted" sigmoid functions, you can only approximate functions that vanish at 0 -- i.e., not all functions. $\endgroup$ – Aryeh May 15 '16 at 13:41
  • $\begingroup$ Do you know something that I can cite about this? $\endgroup$ – mtanti May 17 '16 at 12:34
  • $\begingroup$ You can cite me :) -- more precisely, the simple claim that linear combinations and compositions of functions that vanish at zero must necessarily vanish at zero. $\endgroup$ – Aryeh May 17 '16 at 13:00
  • $\begingroup$ As an expression of gratitude, you can mark my answer "correct". $\endgroup$ – Aryeh May 17 '16 at 13:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.