1
$\begingroup$

I was using MDP on my work to make optimal decision. I used discrete time, finite state MDP. I assumed that I will have an initial parameters, like the Reward/Cost, state transition probabilities and actions, to make the optimal decisions. Then, I used the value iteration algorithm to find the expected values for each possible state under the given MDP configuration.

Here is my problem. As I understand, after getting the optimal policy from Value Iteration Algorithm, the policy will stay valid under the initial MDP configuration. However, I believe the policy must be adjusted whenever there is change on MDP configurations (specifically change in Reward/Cost and Transition probabilities). So, I am looking for the best solution that adjust the policy instead of recalculating the policy again using VIA under the new MDP configuration.

My work assumes the states and action doesn't change, only the cost and transition probability changes. Please suggest me any solution or reference to similar works. Thanks.

$\endgroup$
3
$\begingroup$

The set of optimal policies for one MDP may be completely different than for another MDP even if the two MPDs differ only in the rewards and/or transition probabilities.

One thing you might try in order to save resources in some cases when you change the MDP (e.g. when the set of optimal policies does not change too much) is to use policy iteration to find optimal policies and to start it for the new MDP from an optimal policy for the old MDP. Theoretically, you should bear in mind that there are exponential time examples for policy iteration that start and finish at policies that differ in only one action, but these are pathological examples that will almost certainly not arise in practice.

You may find some relevant information if you search for "sensitivity analysis" or "perturbation analysis" for MDPs.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.