In the context of the following question: off-policy and offline policy reinforcement learning , it can be concluded that off-policy/on-policy learning can be orthogonal to an online/offline sampling scenario.
I am having trouble connecting these concepts to the idea of evaluating an RL approach (target/behavior policy) aimed to be deployed in a real-world environment (e.g. a web application).
In this case, I believe that the online evaluation would be to test the RL approach when deployed and that the offline evaluation would be to use historical data or a simulation of the real-world environment.
Following this assumption, examples of the different evaluation combinations can be:
- Off-policy/offline evaluation: evaluate the target policy using a historical dataset collected with a policy other than the target policy.
- On-policy/online evaluation: experience used to evaluate is collected by following the target policy sampling directly from the real-world environment.
But it is hard for me to think of examples for the other combinations:
- On-policy/offline evaluation: data used to evaluate the target policy is collected by following the target policy on a simulation of the real-world environment. Does this make sense?
- Off-policy/online evaluation: I can't think of an example that makes sense, i.e., if the target policy can be evaluated directly with the real-world environment, why use a different behavior/target policy to sample experience?
Note that my question is in the context of evaluation, i.e. the goal is only to evaluate the target policy and not to learn a new policy.
I am wondering if any of these ideas make sense and if anyone can think of an example for the case of Off-policy/online evaluation?