# Reinforcement Learning and Optimistic Decisions

Sutton recommended using an optimistic heuristic on decision making by considering the upper bound of the confidence range of an action-value. From my testing, it seems to work. I find this fascinating and wonder if it could be proven that optimistic thinking (within these limits) is actually a sound policy for life in general. This may sound a bit off-topic, but I don't think so. I universal truth that can actually be codified would be a real discovery, imho.

Question: is it a universally sound policy to use the "high" confidence probability in the formula:

$ActionValue = P(X) \cdot Value(X)$ <- utility theory, reinforcement learning

So, $P(X)$ will have a confidence interval based on the number of samples. You can optimistically just use $P(X) + confInterval / 2$ to get $Popt(X)$. This seems to work well in practice. Is this a universal truth?

• Yes, following the UCB policy gives $\tilde{O}(\sqrt{T})$ regret, which is asymptotically optimal for the MAB problem. The MAB problem is a special case of reinforcement learning, with a time horizon of $1$. – Lev Reyzin Aug 15 '11 at 15:57