2
$\begingroup$

In the context of the following question: off-policy and offline policy reinforcement learning , it can be concluded that off-policy/on-policy learning can be orthogonal to an online/offline sampling scenario.

I am having trouble connecting these concepts to the idea of evaluating an RL approach (target/behavior policy) aimed to be deployed in a real-world environment (e.g. a web application).

In this case, I believe that the online evaluation would be to test the RL approach when deployed and that the offline evaluation would be to use historical data or a simulation of the real-world environment.

Following this assumption, examples of the different evaluation combinations can be:

  • Off-policy/offline evaluation: evaluate the target policy using a historical dataset collected with a policy other than the target policy.
  • On-policy/online evaluation: experience used to evaluate is collected by following the target policy sampling directly from the real-world environment.

But it is hard for me to think of examples for the other combinations:

  • On-policy/offline evaluation: data used to evaluate the target policy is collected by following the target policy on a simulation of the real-world environment. Does this make sense?
  • Off-policy/online evaluation: I can't think of an example that makes sense, i.e., if the target policy can be evaluated directly with the real-world environment, why use a different behavior/target policy to sample experience?

Note that my question is in the context of evaluation, i.e. the goal is only to evaluate the target policy and not to learn a new policy.

I am wondering if any of these ideas make sense and if anyone can think of an example for the case of Off-policy/online evaluation?

$\endgroup$
1
$\begingroup$

Online/Offline learning

Whether your algorithm learns online or offline is governed by when the agent's experience is able to contribute to its learning.

For example, consider value iteration algorithms for action values. If you use Monte Carlo methods to calculate your value updates, then you'd be performing offline learning as the agent will have to reach the end of an episode to calculate the full MC return for any action value. The full Monte Carlo return and the action value definition (expected return) is given by

$G_t = R_{t+1} + γ R_{t+2} + γ^2 R_{t+3} + … + γ^{T-t-1} R_T$, and

$ q(s,a) = E[R_{t+1} + γ R_{t+2} + γ^2 R_{t+3} + … + γ^{T-t-1} R_T \ |\ S_t = s, A_t = a]$

where $T$ signifies last time step of the episode, and $\gamma$ is the discount factor

On the other hand if you use Temporal Difference methods, the action values can be updated right away by bootstrapping to already calculated values, making the learning online. We use the recursive definition of return to get recursive definition of value (expected return) in TD methods.

$G_t = R_{t+1} + γ G_{t+1}$, and

$ q(s,a) = E[R_{t+1} + γ\ q(S_{t+1}, a') \ |\ S_t = s, A_t = a]$

On-policy/Off-policy learning

This depends on the update function you're using to move your state/action value estimates towards the real expected return of the defintion.

For example, in case of SARSA (update action values to evaluate behavior policy) with 1-step TD updates, we have the update function as

$Q_{t+1}(S_t, A_t) = Q_t(S_t, A_t) + α (R_{t+1} + γ\ Q_t(S_{t+1}, A_{t+1}) - Q_t(S_t, A_t))$

Since this method uses the action value for that action which is actually performed in the next state ($a' = A_{t+1}$ for $q(s,a)$ estimate), the estimates will converge to action values for the behavior policy, and not for the optimal policy. Hence, it is a form of on-policy learning.

Now if you consider Q-learning, the update function is

$Q_{t+1}(S_t, A_t) = Q_t(S_t, A_t) + α (R_{t+1} + γ\ max_{a'} Q_t(S_{t+1}, a') - Q_t(S_t, A_t))$

Here, we don’t need to know what action is to be taken from the next state, $S_{t+1}$. We use the maximum q-value across all actions for the next state to calculate our estimate ($a' = argmax_{a'} Q_t(S_{t+1}, a')$ for $q(s,a)$ estimate). Because of this, the agent learns q-values for the optimal or greedy policy, regardless of the one it is following currently ($A_{t+1}$ doesn’t affect the q-value update). Hence, it is an example of off-policy learning.

Given this information, an example of On-policy/offline evaluation would be simple Monte Carlo learning, where action values are calculated as the average of Monte Carlo returns over multiple episodes.

And an example of Off-policy/online evaluation would be Q-learning with 1-step TD updates. Because of bootstrapping in TD updates, it is online, and due to the nature of the update function (given above), it estimates optimal values.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.