2
$\begingroup$

In a neural network, a bias is a constant term that is added to the weighted input in a neuron/unit: output = activation_function( input1*weight1 + ... + inputn*weightn + bias)

I can see that the bias in a sigmoid activation function adds the ability to control the threshold of the activation. But when I started learning about neural nets we didn't use any bias, just the weighted inputs. I've also been told that sigmoid in particular can do without a bias, but this is not intuitively true for me at all.

So is it true that biases are redundant in sigmoid neural nets? How can neural nets learn to approximate any continuous function if they do not have a bias input?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

No, you don't need a bias. You can have a "dummy" input (input(n+1) in your formualtion) which is always set to 1. Then the bias term is absorbed into the weights.

Improve this answer
$\endgroup$
5
  • $\begingroup$ That's still a bias though. You're just shifting the work on the user instead of the internal logic of the system. $\endgroup$
    – mtanti
    May 15, 2016 at 13:38
  • 6
    $\begingroup$ In that case, it seems that without a bias and using only "unshifted" sigmoid functions, you can only approximate functions that vanish at 0 -- i.e., not all functions. $\endgroup$
    – Aryeh
    May 15, 2016 at 13:41
  • $\begingroup$ Do you know something that I can cite about this? $\endgroup$
    – mtanti
    May 17, 2016 at 12:34
  • $\begingroup$ You can cite me :) -- more precisely, the simple claim that linear combinations and compositions of functions that vanish at zero must necessarily vanish at zero. $\endgroup$
    – Aryeh
    May 17, 2016 at 13:00
  • $\begingroup$ As an expression of gratitude, you can mark my answer "correct". $\endgroup$
    – Aryeh
    May 17, 2016 at 13:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.