4
$\begingroup$

Most implementations of stochastic gradient descent (SGD) rely on floating points. Is there implementations using infinite or finite precision integer arithmetics ?

$\endgroup$
7
  • 1
    $\begingroup$ Is this question about the theory of stochastic gradient descent or about finding a suitable software package? $\endgroup$
    – Lev Reyzin
    Jan 21, 2012 at 4:32
  • $\begingroup$ The former. I'm not looking for a software library for SGD but rather if there is a paper describing a variant of the algorithm using integer arithmetics. $\endgroup$ Jan 21, 2012 at 17:34
  • 2
    $\begingroup$ eh - representing floating point via ints causes all kinds of numerical problems. The "correct" approach is to use adaptively growing (or even infinite) precision as is done in many geometric questions. $\endgroup$ Jan 25, 2012 at 18:09
  • 1
    $\begingroup$ I see. This reminds me why I never went into numerical analysis... $\endgroup$
    – Lev Reyzin
    Jan 25, 2012 at 19:31
  • 1
    $\begingroup$ I am not sure I understand the point of your question. Why not just change the algorithm to use other representations? (like nested intervals?) $\endgroup$
    – Kaveh
    Jan 28, 2012 at 22:53

1 Answer 1

3
+50
$\begingroup$

see eg Neural network training with constrained integer weights Plagianakos, V.P.; Vrahatis, M.N.;

An Integer Recurrent Artificial Neural Network for Classifying Feature Vectors Roelof K Brouwer PEng, Ph

the general idea in various implementations is to represent/approximate real numbers $x \in[0..1]$ as integers in the form $x \cdot 10^m$ where $m$ is some integer exponent.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.