# Reinforcement Learning and Optimistic Decisions

Sutton recommended using an optimistic heuristic on decision making by considering the upper bound of the confidence range of an action-value. From my testing, it seems to work. I find this fascinating and wonder if it could be proven that optimistic thinking (within these limits) is actually a sound policy for life in general. This may sound a bit off-topic, but I don't think so. I universal truth that can actually be codified would be a real discovery, imho.

Question: is it a universally sound policy to use the "high" confidence probability in the formula:

$ActionValue = P(X) \cdot Value(X)$ <- utility theory, reinforcement learning

So, $P(X)$ will have a confidence interval based on the number of samples. You can optimistically just use $P(X) + confInterval / 2$ to get $Popt(X)$. This seems to work well in practice. Is this a universal truth?

## 1 Answer

The main idea of following upper confidence bounds is minimizing regret, that is, the difference between the received reward and the maximum. The paper to read on this is: "Finite-time analysis of the multiarmed bandit problem" by Peter Auer, Nicoló Cesa-Bianchi, and Peter Fischer.

If we interpret life as a Markov Decision Process, then most reinforcement learning theories apply quite well. I.e. we want to keep an open mind, but we want to do what we think is best exponentially more often than exploratory actions.

Of course, interpreting life as an MDP is a bit of a stretch — foresight, for example, has to be interpreted as simulated experience, but I believe one can find quite a lot of wisdom in reinforcement learning and artificial intelligence in general.

• Yes, following the UCB policy gives $\tilde{O}(\sqrt{T})$ regret, which is asymptotically optimal for the MAB problem. The MAB problem is a special case of reinforcement learning, with a time horizon of $1$. – Lev Reyzin Aug 15 '11 at 15:57