Most implementations of stochastic gradient descent (SGD) rely on floating points. Is there implementations using infinite or finite precision integer arithmetics ?

  • 1
    $\begingroup$ Is this question about the theory of stochastic gradient descent or about finding a suitable software package? $\endgroup$ – Lev Reyzin Jan 21 '12 at 4:32
  • $\begingroup$ The former. I'm not looking for a software library for SGD but rather if there is a paper describing a variant of the algorithm using integer arithmetics. $\endgroup$ – Ghassen Hamrouni Jan 21 '12 at 17:34
  • 2
    $\begingroup$ eh - representing floating point via ints causes all kinds of numerical problems. The "correct" approach is to use adaptively growing (or even infinite) precision as is done in many geometric questions. $\endgroup$ – Suresh Venkat Jan 25 '12 at 18:09
  • 1
    $\begingroup$ I see. This reminds me why I never went into numerical analysis... $\endgroup$ – Lev Reyzin Jan 25 '12 at 19:31
  • 1
    $\begingroup$ I am not sure I understand the point of your question. Why not just change the algorithm to use other representations? (like nested intervals?) $\endgroup$ – Kaveh Jan 28 '12 at 22:53

see eg Neural network training with constrained integer weights Plagianakos, V.P.; Vrahatis, M.N.;

An Integer Recurrent Artificial Neural Network for Classifying Feature Vectors Roelof K Brouwer PEng, Ph

the general idea in various implementations is to represent/approximate real numbers $x \in[0..1]$ as integers in the form $x \cdot 10^m$ where $m$ is some integer exponent.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.