In reinforcement learning, with function approximation, a popular cost function is the Mean value error.

enter image description here

This involves a target value V_pi and a current value estimate V_hat. When deriving the update rule for gradient descent learning, people just ignore V_pi-s dependence on the parameters, using a semi-gradient instead of the true gradient. Why is this? Is it difficult to calculate the true gradient?

The problem is not coming from the above equation but rather arise for bootstrapping metod. Sutton's reinforcement learning book (2nd) gave a pretty good explanation to the problem in Chapter 9.3.

The idea here is you cannot obtain true value for V_pi, but you need to get an approximation to it. My understand is when bootstrapping, such as TD method, V_pi depends on w. This will break the assumption that V_pi is indenpendent of w and thus the gradient we get is not the true gradient and we call it semi-gradient. However, if you are not using bootstrapping, such as Monte Carlo, V_pi will be unbias and you will not have this problem and the above will guarantee to converge to a local minimal.

Your Answer


By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.