[IA 11] What is Rationality and how does it relate to a Rational Agent in AI?

In the Spring and Summer of 2025 I had the heady intent of defining Formal Agents, starting with Rationality as this is a term used throughout the domain of AI.

With the leak of Claude Code’s code this seems even more topical, seeing prompts like “please do not break the law” is quite disappointing for the leading AI lab. Jonny (good kind) (@jonny@neuromatch.social) a digital infrastructure researcher covers the analysis here.

Jonny touches on the subject of gambling and it connects through Rationality and a long history of interest in this field.

So, what is Rationality?

I started with 4 main sources back in 2025:

1. Russell and Norvig’s AI a Modern Approach. The conical work, that I touched on in article 3 I felt the need to add the term “Irrational Performance Measure” as well.

2. Jon McCarthy’s thoughts on AI and his 1958 paper “Programs with Common Sense” where he discussed what he called “common sense reasoning”

3. Jon Doyle’s wonderful apology and his PhD thesis, where the section on deliberation and a formalisation of it was particularly interesting.

4. YouTube interviews with Steven Pinker, he has written a book “Rationality: What It Is, Why It Seems Scarce, Why It Matters”. His work didn’t stick with me, other than to say there are many different views worth considering from outside of the Comp Sci/AI domain. He acknowledges the difference between a decision-based approach and how humans actually think.

Irrational Perofmrance Measure?

Irrational Performance Metrics: (My definition to help understand why an agent may act irrationally) Metrics that can lead to undesirable or counterproductive behaviour when used to evaluate agent performance.

This was the entry into Utility. It’s only really in hindsight that I see this, and I think there is a certain formality that could be added to this definition as this, Irrational Performance Measure, is a subjective term.

It’s simply my way of saying, this Rational Agent is meeting it’s targets but it has no Utility for me.

What is Utility and why is it relevant?

Enter the rabbit hole, grab a drink, and get comfy.

It was a fine Summer in 1738 and the Bernoulli cousins where chewing the fat whilst gambling… OK, I’ve no idea about that but I’d like to think that this came from some friendly family conversations and wikipedia covers the St. Petersburg paradox better than I can.

In brief, Nicolaus Bernoulli invented a game in 1713 and Daniel Bernoulli responded in 1738 and opened the door to Expected Utility theory.

The game Nicolaus designed was to answer a question like: “How much would you pay to enter a game were you could win an infintie amount of money (or lose all you put on the table)?”

Daniel’s response was that it is not about the price, rather the usefulness of the outcome to the player:

The determination of the value of an item must not be based on the price, but rather on the utility it yields … There is no doubt that a gain of one thousand ducats is more significant to the pauper than to a rich man though both gain the same amount.

For me, this is relevant to agent capability. One man’s Rational Performance Measure is another man’s Irrational Performance Measure. Claude Code does what it can to get a user base and win big Enterprise Contracts. I do not think I am along in saying that is the Performance Measure that Anthropic has for it’s Coding Agent.

To be clear, that doesn’t mean the Coding Agent has no Utility for me, it does and I use it daily. However it is most definitely not aligned to my intent and the Performance Measures I see Utility in.

The second act - Bayesian vs Frequentist: the Statistians Civil War

Maybe the above has hooked you, maybe not. Next is a brief tour of probabilty over the next 350 years. Specifically how belief and utility are a foundational of rational decision making.

I have thought about writing to convince you Bayesian Statistics is the correct way to view the world, they are the Jedi’s using the force for good, and that Frequentists are the Empire and only use the force for control.

Controversial and I do not wish to cause offense if you are a well versed Frequentist and have a sound counter-argument. However I am a very happy Bayesian and, some friendly banter aside, don’t find the debate useful. So if you firly see the world as a Frequentist, this isn’t for you.

1738: Expected Utility Theory

Daniel Bernoulli: “Specimen Theoriae Novae de Mensura Sortis”

Key section: The St. Petersburg Paradox resolution — diminishing marginal utility.

Utility ≠ price. What something is worth depends on who you are. This is the “one agents’s Rational Performance Measure is another agent’s Irrational Performance Measure” formalised. 288 years ago. History is useful eh!

1763: Bayes' Theorem

Thomas Bayes: “An Essay Towards Solving a Problem in the Doctrine of Chances” (published posthumously by Richard Price)

Key sections: I–V establish the inverse probability framework. Price’s introduction contextualises why.

You can update beliefs with evidence. His goal was a defense of God’s existance, personally I like his attempts, I do not believe in God (despite my upbringing, however I see the utility in his work and in people’s belief in God). Back to the maths - he used uniform priors and treated probability as objective, out there in the world.

1774: Laplacian Revolution

Pierre-Simon Laplace: “Mémoire sur la probabilité des causes par les événemens” (1774, French original)

Key reading: The opening pages where he defines probability relative to observer knowledge.

“Probability is relative to our ignorance and our knowledge.” His work brought in the idea of subjective probability, building on the work of Bayes and refining it to state that probability belongs to the observer, not the event. This is key.

1812-1814: Consolidation of Bayesian Approach

Laplace: “Théorie analytique des probabilités” (1812) and “Essai philosophique sur les probabilités” (1814)

Key chapter: The Rule of Succession (sun rising tomorrow). The Essai is the readable one; the Théorie analytique is the maths.

Calculated odds of the sun rising tomorrow. This is a great experiment to have done, for us it is easy to say the sun will rise tomorrow and we plan accordingly. If it doesn’t we probably will never now it didn’t, I’m sure our plans to take our kids to school or to tidy the house will not be that useful.

However, one day the sun won’t rise. That day is millions, maybe billions of years away when the sun dies. Or may be sooner if there’s something we don’t understand about the universe.

Eitherway, it is fair to say that the probability of the sun rising tomorrow is not 1.

1921: Keynes’s Logical-Relationist Approach

John Maynard Keynes: “A Treatise on Probability” (also on Project Gutenberg)

Key chapters: Part I, Chapters 1–4: logical-relationist view and the Principle of Indifference. Chapter 4 specifically on when indifference is justified — the “actually equal vs treated as equal” question.

He posited the following, bringing logic and the Principle of Indifference into the conversation.

• “Logical-relationist” theory: Probability as logical relation between evidence and hypothesis
• Principle of Indifference: Refinement of Laplace’s insufficient reason principle
• When should we treat probabilities as equal vs. actually equal?

What a question to ask! Calibration…

1926: Ramsey’s Subjective Revolution

Frank Ramsey: “Truth and Probability” (written 1926, published 1931, in “The Foundations of Mathematics and Other Logical Essays”)

Key essay: “Truth and Probability” — the Dutch Book argument is in sections 3–4.

If your beliefs break the probability axioms, someone can construct bets where you always lose (Dutch Book).

So coherent beliefs must follow the rules. But coherent doesn’t mean correct. Calibration…

1937: De Finetti’s Betting Framework

Bruno de Finetti: “La Prévision: ses lois logiques, ses sources subjectives” (English translation in Kyburg & Smokler’s “Studies in Subjective Probability”)

Key reading: The betting framework and Dutch Book proof. Original is in French; the Kyburg translation is the standard English version.

Made probability operational. Your probability IS your betting price. Proved coherent prices must obey the axioms (total 1). But: you could coherently believe a fair coin is 75/25 heads/tails. Nothing in the framework stops you. The arbitrariness problem. Calibration…

1939: Jeffreys’s Geophysical Bayesianism

Harold Jeffreys: “Theory of Probability”

Key chapter: Chapter 1 (Fundamental Notions) for the “probability is not frequency” argument.

“No probability is simply a frequency.” Probability is rational belief, full stop.

1944: Game Theory and Axiomatic Utility

John von Neumann and Oskar Morgenstern: “Theory of Games and Economic Behavior”

Key sections: Chapter 1 and the Appendix on the axiomatic treatment of utility. The axioms are what matter — the game theory is a separate contribution.

Von Neumann & Morgenstern: Axiomatised utility. Rational preferences must be representable by expected utility.

Connected rational choice to both probability and utility.

1954: Savage’s Subjective Bayesian Synthesis

Leonard J. Savage: “The Foundations of Statistics”

Key chapter: Chapter 2 (Personal Probability) — subjective probability from preference axioms and the Sure-Thing Principle. The synthesis of Dutch Book + Utility + Rational Agency.

Savage: The grand synthesis. Subjective probability + utility + rational choice, unified. Dutch Book + Utility Theory + Rational Agency in one framework.

Five points for consideration

1. Laplacian insight

An agent’s probability assessments are relative to what it knows. For an LLM, what it “knows” is its training data and context window. These are its priors — but they’re opaque, unversioned, and unauditable. Can we make priors explicit and controllable? Are they coherent?

2. Ramsey coherence

We don’t know if LLMs are coherent in any formal sense. Attention provides local (not loigical) consistency to the context - tokens relate to nearby tokens - but that’s not Ramsey coherence. The embedding space has no formal logical consistency. The outputs feel coherent, which is the trap. We can’t verify coherence, and we can’t verify calibration. Both are opaque.

Can Embeddings be made logicial coonsistent?

3. De Finetti operational

Beliefs are revealed through actions, not introspection. You can’t ask an LLM “how confident are you?” and trust the answer (logprobs, confidence-accuracy mismatch). You have to test the output, not the reasoning.

4. Utility

Different stakeholders have different utility functions. Anthropic’s performance measure for Claude Code is not yours. This is the Irrational Performance Measure problem. Who calibrates the system and who defines “good enough”? Hint, it should be anyone reading this ;-)

5. De Finetti’s arbitrariness

A system can be internally consistent and systematically wrong. Nothing in the coherence framework prevents this. What provides the external check?