Most implementations of stochastic gradient descent (SGD) rely on floating points. Is there implementations using infinite or finite precision integer arithmetics ?
$\begingroup$
$\endgroup$
7
-
1$\begingroup$ Is this question about the theory of stochastic gradient descent or about finding a suitable software package? $\endgroup$– Lev Reyzin ♦Jan 21, 2012 at 4:32
-
$\begingroup$ The former. I'm not looking for a software library for SGD but rather if there is a paper describing a variant of the algorithm using integer arithmetics. $\endgroup$– Ghassen HamrouniJan 21, 2012 at 17:34
-
2$\begingroup$ eh - representing floating point via ints causes all kinds of numerical problems. The "correct" approach is to use adaptively growing (or even infinite) precision as is done in many geometric questions. $\endgroup$– Suresh VenkatJan 25, 2012 at 18:09
-
1$\begingroup$ I see. This reminds me why I never went into numerical analysis... $\endgroup$– Lev Reyzin ♦Jan 25, 2012 at 19:31
-
1$\begingroup$ I am not sure I understand the point of your question. Why not just change the algorithm to use other representations? (like nested intervals?) $\endgroup$– KavehJan 28, 2012 at 22:53
|
Show 2 more comments
1 Answer
$\begingroup$
$\endgroup$
see eg Neural network training with constrained integer weights Plagianakos, V.P.; Vrahatis, M.N.;
An Integer Recurrent Artificial Neural Network for Classifying Feature Vectors Roelof K Brouwer PEng, Ph
the general idea in various implementations is to represent/approximate real numbers $x \in[0..1]$ as integers in the form $x \cdot 10^m$ where $m$ is some integer exponent.
