1
$\begingroup$

Q-learning is a well-known algorithm in Reinforcement learning which enjoys great empirical success but with insufficient theoretical understanding. In the tabular setting, it is known that if each state-action pair $(s, a), \forall s \in \mathcal{S}, \forall a \in \mathcal{A}$ is visited for infinite number of times.

In the function approximation setting, however, the algorithm is not known to be convergent. There are some counterexamples where it diverges, under various assumptions. I was wondering if Q-learning with nonlinear function approximation could converge if each state-action pair is visited for infinite times. If not, I wonder if there is any convergent variation of Q-learning algorithm that uses nonlinear function approximation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.