Most implementations of stochastic gradient descent (SGD) rely on floating points. Is there implementations using infinite or finite precision integer arithmetics ?
4
$\begingroup$
$\endgroup$
-
1$\begingroup$ Is this question about the theory of stochastic gradient descent or about finding a suitable software package? $\endgroup$ – Lev Reyzin♦ Jan 21 '12 at 4:32
-
$\begingroup$ The former. I'm not looking for a software library for SGD but rather if there is a paper describing a variant of the algorithm using integer arithmetics. $\endgroup$ – Ghassen Hamrouni Jan 21 '12 at 17:34
-
2$\begingroup$ eh - representing floating point via ints causes all kinds of numerical problems. The "correct" approach is to use adaptively growing (or even infinite) precision as is done in many geometric questions. $\endgroup$ – Suresh Venkat Jan 25 '12 at 18:09
-
1$\begingroup$ I see. This reminds me why I never went into numerical analysis... $\endgroup$ – Lev Reyzin♦ Jan 25 '12 at 19:31
-
1$\begingroup$ I am not sure I understand the point of your question. Why not just change the algorithm to use other representations? (like nested intervals?) $\endgroup$ – Kaveh Jan 28 '12 at 22:53
2
+50
$\begingroup$
$\endgroup$
see eg Neural network training with constrained integer weights Plagianakos, V.P.; Vrahatis, M.N.;
An Integer Recurrent Artificial Neural Network for Classifying Feature Vectors Roelof K Brouwer PEng, Ph
the general idea in various implementations is to represent/approximate real numbers $x \in[0..1]$ as integers in the form $x \cdot 10^m$ where $m$ is some integer exponent.
