The off-policy Monte Carlo control algorithm to learn the optimal state-value function $V^*$ is given as follows, which is obtained from Sutton's book.

Off-policy monte carlo contral algorithm.

I have three questions concerning this algorithm:

  1. Why does the algorithm going backward, that is, it iterates from $t = T-1$ downwards to zero in each episode.

  2. The algorithm ensures that $A_t = \pi(S_t)$ for each $t$ otherwise it jumps out of the loop. Are there any additional assumptions to guarantee this algorithm works? How can it ensure $A_t = \pi(S_t)$ as $\{ S_t, A_t, R_t\}$ is generated by policy $\mu$, which is different from $\pi$.

  3. It seems weight $W$ is the importance sampling ratio. But here the target policy $\pi$ changes over time. I think the reason that $W$ is updated by $$ W \leftarrow W \cdot \frac{1}{ \mu(A_t | S_t)} $$ is because the target policy is always greedy with respect to $Q(s,a)$, which makes $\pi(A_t | S_t) = 1$. Is this correct?

  • $\begingroup$ You should ask these questions on Artificial Intelligence Stack Exchange. Furthermore, please, if you decide to ask them there, ask 1 question per post, even if they are related. It's better to separate the questions so that users can focus on one question per post. $\endgroup$ – user34637 May 5 '20 at 17:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.