Writing
Notes on evaluating agents, simulation, and eval infrastructure.
Essays
- Jun 25, 2026Eval-Driven DevelopmentAgents move decisions out of control flow and into the model. That part of the specification can only be recovered by watching the system act, which makes the eval the place where it gets written.
- Dec 6, 2022Priors for a portfolioA model's job is to be honest about what it doesn't know, in a form a decision can use. Three years of simulating clinical trials taught me what that means.
Notes
- Aug 18, 2026Which branch would you bet onA leaderboard reports scores. A decision needs a preference and an uncertainty, and frontier scenarios are exactly where you have neither by default.
- May 19, 2026Known-bad candidatesAn eval that has never been shown a wrong answer has not been shown to measure anything.
- Apr 14, 2026Five of five is not a statisticWhat repeated eval runs can tell you, what they cannot, and why the two get confused.
Elsewhere
- Apr 2024
- Apr 2024
- Oct 2023
- Jun 2023
- Nov 2022
- Jan 2021When AI Gets it WrongTowards Data Science
- Nov 2020When Data Science Fails UsTowards Data Science
- Jun 2020Time for Sustainable DevelopmentTowards Data Science
- Jun 2019Towards Human Centered EducationTowards Data Science
- Aug 2018Neural Networks for Music GenerationTowards Data Science