110. Finally, Someone Made an Infra-Bayesian RL Agent

(Epistemic status: I mentored the SPAR stream that did all of this, pitched the project in the first place, and contributed project management and much of the understanding of the theory, which makes me exactly as impartial as that sentence implies; audit accordingly. This is meant as a companion post and reference explainer for the stream's talk Friday morning as delivered by FL; read it before or after, or refer to it during - whichever suits you best. I've deliberately left out much of the formal math here, saving that for the talk slides and more formal posts and papers that give a more in-depth treatment of the subject; there's pointers at the end for anyone who wants to go digging. For my entire SPAR stream, all fourteen of my first-coauthors: MA, FA, SMA, AB, SSMJ, AL, CL, AL (a different one), FL (obviously), RR, PR (a different one), ER (who is at the conference!), NT, and MPdV. I am unspeakably proud of all of you, and I could never have made Project Newcomb finally happen without you.)

 

You may recall from last November's "52. Why Infrabayesian Epistemics Should Permit Winning At Causally-Weird Decision Theory Puzzles" that I complained about how the infra-Bayesian research community seemed bizarrely uninterested in implementational details. Whenever I talked to them, it was consistently all regret bounds and no running code, with the question of whether the gap between math and code could be bridged in the name of empiricism waved off whenever I raised it as uninteresting, already solved, "that thing NB abandoned a few years ago, I think?", or in one notable case, as direly dangerous capabilities research. I am pleased to report that my complaint has been comprehensively answered, and that I can claim precisely as much cfredit as that sentence implies: on my friend PR's recommendation (not the one on the team), I pitched the project to SPAR as a project for the Spring 2025 cohort, a team of scholars applied, I decided to accept as many of them as would accept me back, and the gloriously sprawling team went and built the thing. There is now a proof-of-concept infra-Bayesian reinforcement learning agent, complete with experiments, up on arXiv.

It one-boxes.

To understand why that's such a big deal, we need to start with the reason anyone would want such a creature. The classical picture of an ideal reinforcement learning agent is clean enough to slurp Soylent off of: enumerate all possible environments, assign them prior probabilities, act so as to maximize expected reward weighted by credences, push a posterior by Bayesian updating after each observation, converge eventually to the truth, and live optimally ever after. Here's the snag: every guarantee in that sentence hangs from a single assumption so habitual you've stopped seeing it: realizability. In order for any of this to work at all, the true environment has to live somewhere in your hypothesis space. Under misspecification (also known as nonrealizability, depending on your field), not only does this whole apparatus not fail gracefully, but instead fall helplessly prey to the Bayesian's cardinal sin: assigning probability 0 to the right answer. A RL agent which leaves the true environment off its list can be confidently, permanently wrong... and confidently wrong agents can take arbitrarily bad actions with pride.

It gets worse. Misspecification isn't the exception - it's the default, for several nauseatingly good reasons. (Please imagine me ticking the following off on my fingers.) The kinds of agent we care  most about are embedded: they live inside their environments, and so the environment contains the agent itself, and nothing can hold a complete model of something containing it; every hypothesis it could write down is necessarily false. They also exist in policy-dependent environments, where the universe really does care about what you're up to: predictors, adversarial agents, even agents trying to be helpful. They also sometimes exist under conditions of Knightian uncertainty: they enter not a clean gameboard, but a messy and shadowed world already in motion. Classical RL inherently relies on learnability, ratifiability, and the policy-independence of its environment, and thus under conditions such as the foregoing - Knightian uncertainty, predictors, adversarial agents, embeddedness - it does poorly. That is: under the conditions that generally obtain, where anything in the world is in fact paying attention to you, classical RL agents often fail catastrophically; whenever the environment plays back.

So how does infra-Bayesianism propose to handle these everyday troubles? Its answer is to redefine what a hypothesis even is. A Bayesian hypothesis says, roughly, "the world is exactly X" - a complete description, or at least a set of neatly nested descriptions. An infra-Bayesian hypothesis says, yet more roughly, "the world obeys law L" - a partial description, or more formally a full-flavor set of complete or partial ones, where we no longer require any commonality, nor do we admit flattening to sum up total probabilities. Realizability now requires only that any statement in the IBRL agent's head be true, not that the entire world fit inside of it. The agent still keeps ordinary probabilities across its candidate laws, but within any given such law, it holds the remaining uncertainty over those possible worlds as Knightian - no distribution, and no real attempt to learn one, with the set treated much the way you might reason about a choice already made by an adversary. (Historically, I've called that adversary the Demiurge; Its literature name is Murphy. Our team's slides just say "adversary", which is more professional but less fun.)

For an expected value operator, we don't take fully weighted averages - for what would that even mean, in a world of Knightian uncertainty? - but rather evaluate every policy on their worst case within each law, and only then weigh across those laws by ordinary probability. Then we choose the best option under that measure, with the whole process notated thus: \(\pi^* = \arg\max_{\pi} \sum_i P(\Psi_i) \min_{a \in \Psi_i} \mathbb{E}_a(\pi)\). We update similarly: only ever eliminate what you can tell for sure is now impossible, and reweight possible laws accordingly. Put in plainer English: it's a matter of pessimism exactly where the agent cannot learn, and Bayes everywhere it can.

Here's a worked example: consider the case of a coin with unknown bias, with probability of heads anywhere from 0 to 1. With bias drawn randomly just once, or even a few times, we have the kind of classical uncertainty a classical RL agent can handle: watch, learn, bet when at last the posterior clears some choice of bar. But a coin chosen freshly each round by someone that can read your source code must be Knightian: an observation teaches you nothing about the next round, the worst case is a coin that starts coming up against you precisely when you feel brave enough to start betting, and the only correct move is to walk away. The Bayesian keeps a posterior - the infra-Bayesian keeps its money.

A picture worth keeping in your head is a spectrum: at one end, the pure pessimist. It has just one trivially-true hypothesis, never learns, and always takes the least-bad action it knew of at the start; it practices maximal paranoia and thus reaps minimal performance. On the other extreme is the pure Bayesian: rich hypotheses, strong guarantees, and the kind of brittleness that often comes with strength - catastrophic failure when it goes wrong. Between them, the infra-Bayesian agent interpolates, and best of all, degrades gracefully, discarding law after law as the world falsifies them, buying performance with added bookkeeping and caution and refunding it honestly when an assumption bounces. It strives not for the best answers, but for least-wrong answers, as best it can manage.

A peek at the machinery for only those who want it: those laws are sets of "a-measures" or "affine measures", written as ordered pairs \(\lambda \mu , b)\): a scaled probability measure plus a scalar offset that tracks how well you're doing on the counterfactual branches you've updated away from, because that pessimism has an appropriately long and wary time horizon; expectation, as above, is taken as a minimum over the resulting set. That offset is actually necessary: it's what keeps the agent dynamically consistent rather than regret its own predictable updates. The team's implementation actually only needed the basics - infradistributions, the pessimistic expectation operator, and some off-history bookkeeping - and never touched the heavier gear: no history gluing, no multi-step infra-Bayesian updates, no surcausal hypothese, and - we'll get to it - no Nirvana trick.

Alright, enough about the background theory. What did the team actually build? We kept our ambitions low, knowing that a scant three months goes faster than you'd think, especially with everyone remote, and it's much better for morale to aim for minimum viable results and go further if there's more time rather than shoot for the moon and die in deep space. We went for stateless environments, discrete action and observation spaces, and finite outcomes, all chosen so that the theory's hardest moving parts could all be simplified honestly or dodged outright rather than fudged. Three implementation moves carried most of the weight, each of them satisfying in that ineffable way that good engineering against infinite objects often is.

• Strictly speaking, an infradistribution is an uncountable set of a-measures, but expectation is a minimum which only extremal points can ever achieve. So we store those finitely many points to recompute from and can trash the rest lightheartedly.
• A stateless environment means history's just noise, so the agent's entire memory per action can be a histogram of outcomes rather than a mess of strings.
• The measures themselves can thus be constructed on the fly, because the history tree would otherwise grow exponentially, and writing all down is for people with more compute budget than sense.

One consequence I want on the record: in this machine, the policy-dependence machinery ends up simplified so far that the team never had to invoke the Nirvana trick. The Demiurge went the whole project fully clothed. 

So what did our world-first toy IBRL agent do? The tl;dr is "everything you would obviously want, and everything you'd reasonably hope for".

• To start with, the result you should want to be boring. If you hand the infra-Bayesian agent a world with purely classical hypotheses, each law a single world, it'll agree with the classical Bayesian agent precisely, play for play, regret curve for regret curve. Recall that infra-Bayesianism is a strict extension, not a rival that loses on home turf or pays paranoia costs when it need not. If this result had failed, nothing else would matter. Thankfully, the bread course passed.
• Then there's the Knightian bandit. The setup, in brief - two arms, with payout probabilities chosen by an adversary confined to known intervals: \(p_1 \in [0.3, 0.7], p_2 \in [0.4, 0.8]\). The worst corner of the box - (0.3, 0.4) - has arm 2 ahead, so the IB agent, ever evaluating on worst cases, plays arm 2 and stays there, while the classical agent's fate depends entirely on its prior, and in a world with an adversary, it gets its lunch eaten by worst-case regret. In our research diary, the team notes - correctly, and in print - that an ordinary robust-RL agent could likely have solved this one, too. That's the exact correct amount of self-suspicion, but the Knightian bandit was a pretty facile case anyway, and between those two reasons, that's why this experiment is the apppetizer rather than the main course.
• That main course? The decision-theory zoo. We need a predictor for this one, but we made sure that the predictor is modelled honestly: it samples its prediction from the agent's policy, independent of the subsequently sampled action and of any randomization it involves, and that way no causal structure gets quietly broken; and at any rate, in the stateless setting, a policy is just a distribution over actions, so we can score the agent's discretized policies directly against the predictor. This meant we were doing an honest rendition of the kind of logical-time loopiness decision theory puzzles call for. This also meant it was worth testing the IB agent against a variety of varyingly accurate predictors - everything from the maximally oracular right on down to "flip a coin and pray".
• On Newcomb's problem - the marquee weird decision theory puzzle! - the agent tracks the optimal reward curve against every predictor, two-boxing against the coin-flipping fraud and switching to one-boxing the moment the predictors are accurate to make it pay. Recall that classical RL agents, causal decision theorists to the last, famously argue themselves out of the money here; our IB agent, run rather than argued about, makes it rain.
• On Asymmetric Death in Damascus - where Death knows your policy, and you'd prefer living to dying at home to dying abroad - our IB agent lands the optimal mixed strategy, just as hoped.
• These are the puzzles I hand-waved about back in November; the ones where extra information can hurt you and causality twists itself into knots. I'm not hand-waving anymore; I'm waving regret plots.
• Finally, for dessert, the experiment that justifies the spectrum picture at top. Give the IB agent three nested hypotheses about the predictor it's up against: "there is no predictor and you can be bold", "the predictor is weak", and the trivially-true pessimistic belief in a potentially-perfect predictor - and then run the thing in universes that break those laws one by one. Where the sharp hypothesis holds, it performs like a Bayesian. Where the weak predictor is still a live possibility, it falls back to the partial law and the performance that that law can support. And where paranoia is the only thing it can trust, it retreats, after an initial sad-but-honest learning phase, to principled pessimism. That's graceful failure observed in a running system rather than asserted on a slide, and as far as I'm concerned, it's the project's true headline: not that maximalist pessimism always wins, but that calibrated and principled pessimism loses slowly, retreating in good order, giving up only what it must.

One last thing that the talk won't have covered but that you surely came for: why do we care so much about infra-Bayesianism for AI alignment? Just like the IB agent picking between bold and cautious, we've got two branches, one modest and one ambitious.

• The modest branch: an agent that evaluates on worst cases within what it concedes it can't learn won't take actions carrying a massive downside risk, if it turns out to be wrong about what world it's really in. If you want cautious optimization, there it is, right there as a structural property of the way an IB agent's decision rule works - not as a bolted-on tax you'll one day wake to find has been optimized away.
• The ambitious branch: an AI built with IB epistemology in its very chips takes seriously - in its type signature! - the possibility that it might be fundamentally wrong, on a meta as well as object level... and potentially wrong in particular about the things we most need it to be humble about - namely, human values, whatever those turn out to be, if indeed they're a single real thing at all.

And beneath both branches: infra-Bayesianism is one of the very few frameworks in which embedded agency isn't a paradox but rather a modelling assumption; the world being stranger than your hypothesis class might permit is the starting point of the formalism, not its failure mode. (The full research program this fits into is the "learning theoretic agenda", with VK and A"D"A's sequences the source, complete with worked examples.)

Every good research paper admits its shortcomings and longs for future directions; this blog post pays such tribute.

On the infra-Bayesian theory side, there's the missed wins problem - an ideal infra-Bayesian agent can be so pessimistic that "why doesn't it just shut itself off immediately" is a live research question, not a gotcha. Counterfactuals remain genuinely awkward, mostly for decision theory puzzle reasons; thankfully, an IB agent doesn't shut down just because it might get taken out by a stray meteorite - it just gives up on those universes in advance, when it comes time to update. "Human values" as a single referent may simply be meaningless, in which case the ambitious branch is aimed at fog, but that's just life. Whether Knightian uncertainty is even real - whether there's any principled line separating "cannot learn" from "have not learned yet" - is philosophically contested, and this entire edifice cares at least a little about the answer. And the Nirvana trick, which the full theory leans on for policy dependence, was avoided here, not vindicated; its reasonableness is still an open bill... though I think it's probably fine.

On the empirical side, there is so much more we could have done, if we'd had more time, more funding, more energy, and greater commitment. This is ultimately still just a stateless proof-of-concept, where one of its two headline environments falls to ordinary robust RL, and we never quite got around to implementing a principled exploration strategy; we were thinking of things like Thompson sampling, upper confidence bound sampling, and epsilon-exploration. There is so much higher to build... but at least now, the tower has a ground floor you stand on, and that's one more floor that it had when I wrote in November, and the move from zero to one is fammously the hardest and most impactful of them all.

My next complaints are already drafted, and I'm sure you can predict the tune: the need for multi-state environments, a real exploration strategy, actual histories and gluing and proper updates, the Nirvana trick faced rather than dodged. If the implementational details bore you, you now know fifteen people who disagree, and we have a repository to show for it. And if you'd rather argue with the theory than run it: as of this spring, the theory is something you can run. That excuse, at least, is retired.

If you want to look more deeply into all this:

• The paper: "Infra-Bayesian Reinforcement Learning Agents Outperform Classical RL For Worst-Case Robustness", https://arxiv.org/abs/2605.23146
• The code: our repo is at github.com/SPAR-S26-IBRL-Stream/Project-Newcomb . The environments are small enough to read in an afternoon, rerun in an evening, and extend by night.
• The talk: if you're not in Berkeley tomorrow morning in Rat Park at 11am sharp, sucks to be you. Be there or miss out.
• The deep end: go looking on LessWrong for far more than you ever wanted to read on infra-Bayesianism. There's sequences galore; brush up on your measure theory, and you can find my old distillation if you try.