Q-learning is a well-known algorithm in Reinforcement learning which enjoys great empirical success but with insufficient theoretical understanding. In the tabular setting, it is known that if each state-action pair $(s, a), \forall s \in \mathcal{S}, \forall a \in \mathcal{A}$ is visited for infinite number of times.

In the function approximation setting, however, the algorithm is not known to be convergent. There are some counterexamples where it diverges, under various assumptions. I was wondering if Q-learning with nonlinear function approximation could converge if each state-action pair is visited for infinite times. If not, I wonder if there is any convergent variation of Q-learning algorithm that uses nonlinear function approximation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.