so for the people who are not versed with reinforcement learning much let me explain the multi armed bandit problem simply

imagine you walk into a casino to multiple slot machines and each of them holds a reward which you can achieve. now the problem at hand is what is the best strategy that you can maximize your rewards or minimize your loss in the long run? will you keep on playing on one slot machine forever hoping that it’s the best choice or will you try other machines as well thinking what if they might give a better return?

now let’s think of each machine as a “choice” → career paths, relationships, projects, moving cities, random ideas still lying in your notes app, those gut wrenching thoughts you have at 3 am

you don’t know which choice would give the best “reward” and in most cases you don’t even know what the reward is

but you can’t keep on pulling the same machine because it worked once, you’ve gotta explore, you’ve gotta risk it and it may happen that you might waste a few pulls but what if you get a return so asymmetric you never expected of?

this is where “optimism in the face of uncertainty” comes in → when you don’t know, assume the unknown might be better, so act if the unexplored could surprise you

formally, for each action $a$, there’s a value estimate $Q(a)$, and the update for that is:

$$ Q(a)←Q(a)+α(r−Q(a)) $$

where $α$ is the learning rate and $r$ is the reward you get from pulling that arm

now the main crux of this approach is that you initialize all $Q(a)$ values to something optimistically high. let $Q(a) = 10$ or $Q(a) = \infty$ if you’re feeling lucky for all the $a$ at the start

this will create a bias for the agent (referring to “us” here) to explore actions that it hasn’t tried yet, because the unvisited actions would look good and would feel full of hope

<aside> 🦝

this approach would keep you curious

</aside>

now the main issue with humans is that we love to think in binary, right vs wrong, good vs bad, leave or stay, etc

it’s kinda easier that way

but in reality there’s always a gray area because there’s no fixed horizon of life or no specified pre-determined path which you can follow to achieve what you desire

so what’s the smart thing to do?

<aside> 🏙️

act as if the unknown, unconquered might surprise you

now if the end result is good or bad that’s a story for later

but be optimistic not because you’re naive but mathematically or statistically the unexplored may still be better

</aside>