Sutton recommended using an optimistic heuristic on decision making by considering the upper bound of the confidence range of an action-value. From my testing, it seems to work. I find this fascinating and wonder if it could be proven that optimistic thinking (within these limits) is actually a sound policy for life in general. This may sound a bit off-topic, but I don't think so. I universal truth that can actually be codified would be a real discovery, imho.
Question: is it a universally sound policy to use the "high" confidence probability in the formula:
$ActionValue = P(X) \cdot Value(X)$ <- utility theory, reinforcement learning
So, $P(X)$ will have a confidence interval based on the number of samples. You can optimistically just use $P(X) + confInterval / 2$ to get $Popt(X)$. This seems to work well in practice. Is this a universal truth?