107. If We Had a Mature Theory of Agent Foundations, How Would We Know?
(Epistemic status: A vital question people don't seem to have asked themselves. A thought experiment run in public, though the credences are soft enough to spread on toast and the butter is closer to butterfly. Roughly a third of me suspects the central question below is malformed; that mass is doing honest work in every paragraph, and I won't pretend otherwise. Published while I'm at a conference: if you have errata, quibbles, or improvements for me, bring them.) Suppose you woke up tomorrow to find, there on your desk, an elegant but battered tome: a three-hundred-page brick titled "A Theory of Agent Foundations". Suppose it fell out of the sky: the perfect solution to correctly paradigmatize agent foundations. What an agent is; what a goal is; what values are, and what human values are and should be; how to detect all of those; measure them; taxonomize and typify and employ them. Suppose all of that were simply there, neither requiring or offering any explan...