I've been reading some interesting papers recently on methods for automatically and adaptively setting the learning rate in stochastic gradient descent (SGD). In particular, "No more pesky learning rate" and "Adaptive learning rates and parallelization for stochastic, sparse, non-smooth gradients" --- which are a pair of related papers that deal with a method for automatically adjusting the learning rate during stochastic gradient descent. The first paper introduces the basic method and the second paper talks mostly about how to apply it in practical settings where one typically uses mini-batches and may have a sparse gradient etc.

I was wondering if a similar approach exists for the "online EM algorithm", where one would be learning an adaptive forgetting factor rather than an adaptive learning rate. Clearly, there is a relationship between the two learning approaches, but it seems like more effort has been put into (and thus more results obtained) SGD. Are there similar methods to rid the online EM of it's meta-parameter? Where could I find some publications on this topic?




Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.