Something Different This Way Comes

In which I talk about winning—Nothing ends, Adrian—Payouts are for games and death

Defining Winning

Having spent so long on my analysis of rationalism, maybe it’s time we tackled a little bit of definitional word-play. What the hell do we mean when we say “winning”, anyways? I refer again to original sources:

As we established in the previous sections on game theory, winning is a state achievable in a game when you attain the illusory goal defined by the game within its context, while using the agency afforded to you by the game. Yudkowsky, it seems, agrees with my definition. The origin of the phrase “rationalists should win”, as one might expect, comes from bounded thought experiments with clear and well-defined payoffs that can be compared against each other. Winning, then, is achieving the optimal payoff in that scenario. Yudkowsky then extends this to say that, if there is a general, systematised way of winning, that way would be rationality.

Allow me then to reformulate this definition slightly: Rationality is, over an extended period of time, achieving the desired payouts from a series of games ¹.

Games are Discrete, Life is Continuous

Seeing that definition, it might be easy to accuse me of being unfair. After all, Yudkowsky et al. do not limit themselves to the realm of games, they expect their method to generalise to the whole of life, with its myriad situations and complexities. Of course, I have already made my case as to why games generalise poorly to real life situations, and why grand systems of universal understanding are a poor fit for real life as well. However, for the purposes of this interlude on winning, let us take them at their word.

Let us say that rationality works. That is to say, given a series of policies and their corresponding context (I also yield them this element of the doubt, because rationalism is path-dependent), rationalism allows us to select with very high accuracy the policy with the most desired payoff, as specified by the rationalist. Let us say, in short, that rationalists win, systematically. I claim that this is still a flawed form of thinking about the world. This is because the world is continuous, while rationalism is discrete.

Perhaps it would be easier to help explain with an example. In the seminal comic Watchmen, the villain/self-proclaimed hero Ozymandias (real name Adrian Veidt) creates a fake alien monster and uses it to kill New York City in a fake alien invasion. He does this to end the cold war, which he sees as an existential threat to humanity. His plan works—stunned by the emergence of a tentacled monster, the US and the Soviet Union immediately agree to cease hostilities and defend against the imagined alien threat. Leaving aside issues of morality and consent, in the standard utilitarian calculus Ozymandias has traded the lives of millions to save billions. He has, unmistakeably, won.

Then, at the very end of the comic, he talks to the superhero Dr. Manhattan, a being who can see into the past and future and also essentially manipulate matter at the molecular level. For all intents and purposes, he is talking to god. And he lays out his case, that his sacrifice of a city has enabled the aversion of a nuclear catastrophe which would have ended humanity. He asks if it was all worth it, “in the end”.

The End is Never the End

Most people would hardly call what Ozymandias did a foolproof plan. If word got out that the alien was faked, not only would he be the world’s most wanted mass murderer, the cold war would probably start up again. Yet, in his head, the game ended when the monster exploded onto the scene and the two governments joined hands. Because he thought in comic-book logic, in terms of stories and plots and schemes with grand finales, Ozymandias convinced himself that he was making a one-time trade off of a great number of lives to save an even greater number of lives, when in reality he killed millions for what might turn out to be an extremely short-lived truce.

Therein lies the problem with “winning” as a concept. To win, there must be an end state where we cash out, some outcome we accept as “the desirable ending”. We get the girl, we beat the boss, we achieve our weight loss goals, we publish the paper. Except, of course, life continues on after we optimise our way to our goals (unless we die). And we are never so perfect in our foresight as to never be surprised when we get what we think we want.

Indeed, in the worst case scenario the continuity of life can come back and rob us of “wins” we thought had been “cashed out” already: if you win in a blackjack game, bet it all again, and lose it, not only have you lost your original pot, you’ve also lost those winnings you thought were yours already. If you do badly after getting promoted, you might get fired, losing both your newly expanded salary and your original salary.

Just Keep Winning

It is possible to argue that a rationalist would not do what Ozymandias did, or at least that a rationalist would simply not allow the truce to devolve back into war or allow their plot be discovered. After all, a rationalist is into systematic winning, not big one-off gambles. So long as they consistently make optimal choices, surely a rationalist can avoid unwanted outcomes that emerge from short-term success?

This might be true if rationalism was perfect and infallible, if it was an invincible method of always getting what you wanted. However, if you accept that rationalism is tethered to reality at all (as Yudkowsky advises in the above article), then you can also see ways in which temporarily achieving “win states” might, over a longer period, manoeuvre yourself into a worse position, and eventually cut off all possibilities of success. You manage to socially engineer your way into the tech company HQ to talk to your idol, and he calls the cops on you for breaking and entering. You attract loads of funding for a new startup that ends up being untenable, leaving you with angry debtors and no revenue stream. You beat the AI at Newcomb’s Paradox, and it decides to torture you in a simulation as revenge.

Directions over Goals

So what are we to do? There are obviously certain situations where thinking about problems in terms of fixed end states is optimal. Bounded games of risk like blackjack at a casino have a very clear game and a set of payouts that end the game ². Any situation where you might die offers a very definite and negative final payout as a possibility you ought to consider carefully. But what about the rest of life, when things are muddy and uncertain, and where nothing seems to end when you want it to?

Heuristics come in again. It can help to loosen your grip on what constitutes as “winning” to be more statistical: what matters is not the specific instance of victory, but more the overall trend of movements towards some desired direction. The candy bar you eat did not destroy your weight loss regimen so long as you make up for it by going to the gym regularly. Thus, winning is not defined as getting the desired outcomes in a series of games, but exercising the option to move in a desired direction consistently. We change the focus from getting an outcome you want to the exercise of choice in a direction you want to head.

Most importantly, this means accepting that sometimes you will “lose”, and that’s okay: always getting what you want may not actually be good for you. Small failures can offer much more interesting information that lucky breaks. Perhaps this sounds like loser-talk, like the sour grapes of someone who can’t get their way. But I would venture to say that a lot of the times “getting your way” is not in your control at all. Sometimes the book you want has already been sold. Sometimes the casino game you’re playing is rigged. A looser grip on the idea of “fairness” is a critical adaptive tool in a cruel and capricious world.

End Interlude

Anyways, I hope that clarifies a little bit of what I meant in the previous section. On to the next section, where I try to expand these principles into something resembling a political program.