Essays and notes on evaluating agents, simulation, and eval infrastructure.https://andy.spezzatti.com/https://andy.spezzatti.com/notes/which-branch-would-you-bet-on/https://andy.spezzatti.com/notes/which-branch-would-you-bet-on/A leaderboard reports scores. A decision needs a preference and an uncertainty, and frontier scenarios are exactly where you have neither by default.Tue, 18 Aug 2026 00:00:00 GMThttps://andy.spezzatti.com/writing/eval-driven-development/https://andy.spezzatti.com/writing/eval-driven-development/Agents move decisions out of control flow and into the model. That part of the specification can only be recovered by watching the system act, which makes the eval the place where it gets written.Thu, 25 Jun 2026 00:00:00 GMThttps://andy.spezzatti.com/notes/known-bad-candidates/https://andy.spezzatti.com/notes/known-bad-candidates/An eval that has never been shown a wrong answer has not been shown to measure anything.Tue, 19 May 2026 00:00:00 GMThttps://andy.spezzatti.com/notes/five-of-five-is-not-a-statistic/https://andy.spezzatti.com/notes/five-of-five-is-not-a-statistic/What repeated eval runs can tell you, what they cannot, and why the two get confused.Tue, 14 Apr 2026 00:00:00 GMThttps://andy.spezzatti.com/writing/priors-for-a-portfolio/https://andy.spezzatti.com/writing/priors-for-a-portfolio/A model's job is to be honest about what it doesn't know, in a form a decision can use. Three years of simulating clinical trials taught me what that means.Tue, 06 Dec 2022 00:00:00 GMT