In reinforcement learning, with function approximation, a popular cost function is the Mean value error.
This involves a target value V_pi and a current value estimate V_hat. When deriving the update rule for gradient descent learning, people just ignore V_pi-s dependence on the parameters, using a semi-gradient instead of the true gradient. Why is this? Is it difficult to calculate the true gradient?