Priors for a portfolio

Deciding where to run a clinical trial is a bet placed years before the answer arrives. Choose the countries, the sites, the number of each, and the order in which to open them, and the consequences show up as enrollment curves eighteen months later. Open too few sites and the trial runs long; every month of delay is a month patients wait for a treatment that may already work. Open too many and you have staffed and paid for sites that enroll nobody. A large company runs hundreds of these bets at once, and they compete for the same people, the same countries, and the same budget.

I spent three years at Genentech building the models this planning ran on. What I expected to learn was statistics. What I actually learned was what a model owes the people who have to act on it.

Two patients at a site

The data problem is that almost every unit you care about is sparse. A trial might have a hundred and fifty sites. Three months in, one has enrolled two patients, one has enrolled nine, most have enrolled zero or one. The question the planner asks is simple: how many patients will this site contribute by the end, and should we open another like it? The naive estimate from two patients in three months is worthless, and everyone knows it is worthless, so the temptation is to ignore the site data and use a country average instead.

Both answers are wrong in the same way. They decide, in advance and by fiat, how much this site resembles other sites. The estimate from two patients says it resembles nobody. The country average says it resembles everyone in the country equally. Neither is a claim anyone would defend if asked to state it out loud.

A hierarchical prior is a way of stating it out loud. The enrollment rate at each site is drawn from a distribution over sites in that country and indication; that distribution has its own parameters, learned from every site the company has ever run there. A site with two patients starts out believing what the population believes, and moves toward its own rate exactly as fast as its own data justifies. Nine patients in three months pulls hard. Two patients pulls a little. Zero patients in a site that opened last week pulls not at all, because the model knows the site has not yet had time to tell you anything.¹

The mathematics here is old and not exotic. Poisson arrivals with a Gamma prior on the rate give you a closed-form posterior and a negative-binomial predictive distribution, and the literature on using this for multicenter recruitment was mature before I arrived.² What was not mature, in the organization, was the idea underneath it: that a prior is a bet you would be willing to make before seeing the data, that a hierarchy is a statement about who resembles whom, and that both of those are decisions a human is making whether or not they write them down. Writing them down was most of the work.

The decision is the output

The instinct of anyone who builds a forecasting model is to report the forecast. Expected enrollment completion: March of next year. That number is almost useless to the person deciding whether to open sites in Poland.

What they need is the shape of what could happen and how it responds to what they do. If we add six sites in two countries, how much does the tail of the completion date move? Which plan is still acceptable if the optimistic sites turn out to be ordinary? The answer is not a date; it is a comparison of distributions under alternative plans, and the thing being chosen is the plan, not the estimate.

This changes what the model is for. It is not a crystal ball to be consulted and then set aside. It is the arena in which plans are compared, and its job is to be honest about the spread of outcomes under each one. A model that reports a confident date and is wrong has failed twice: once on the date, and once by hiding that the date was never the point.

The costs were asymmetric, which made point estimates actively dangerous. A site that never enrolls costs months of setup and a fixed budget. A trial that runs a year long costs a year of a drug’s patent life and a year of patients without it. Ranking plans by expected completion date treats those as the same kind of error. Ranking them by the probability of missing a deadline that matters, under a budget, does not. The country-selection framework we built did the second thing, and its output was a ranked set of plans with the uncertainty attached, not a recommendation and a number.

Why simulate

If the posterior is closed-form, why simulate anything? Because the questions were about paths and constraints, not about rates.

A trial is a schedule of site activations subject to regulatory lead times that differ by country, monitors who can each cover so many sites, budgets that release in tranches, and a portfolio of other trials competing for the same monitors and the same countries. The completion date of one study depends on a decision made in a different study about a team in a third country. There is no closed form for that. There is a simulation: draw enrollment rates from the posterior, activate sites according to the plan and its constraints, watch patients arrive, and record what happens. Then do it ten thousand times.

Discrete-event simulation is how the posterior gets to act in the world. The Bayesian machinery says what each site might do; the simulation says what the portfolio does when you let those sites interact with lead times, staffing limits, and each other. Neither is useful alone. A distribution over rates with no operational model tells you nothing about staffing. An operational model with point-estimate rates gives you a confident schedule that is wrong in a way you cannot see.

Staffing was where this paid off first. Given a portfolio plan, how many monitors, of which functions, in which countries, in which quarter? The simulation produced it as a byproduct: every run left behind a record of who was needed where and when, and the distribution over those records was the staffing plan with its uncertainty. That was the output leadership had never had, because nobody could have built it by hand.

Legible uncertainty

None of this matters if the person deciding cannot read it.

The hardest part of the work was not the model. It was finding a form in which a distribution over outcomes could be consumed by someone with twenty minutes and a decision to make. A density plot is honest and unreadable. A single date is readable and dishonest. What worked, mostly, was probabilities of specific things: the chance of finishing by the date that mattered, under each plan, and how that chance moved when the plan changed. People make decisions about those. They do not make decisions about credible intervals.

There is a temptation, when the audience wants a number, to give them one and keep the uncertainty for yourself. It is a mistake, and not only for the obvious reason. When the number is wrong, and it will be, the modeler has taught the organization that the model is wrong, rather than that the world was uncertain and the model said so. The uncertainty is the model’s most important output. Hiding it converts every miss into a loss of trust.

Then 2020 happened, and every enrollment assumption in the portfolio broke at once. Sites closed. Countries stopped accepting new studies. Monitors could not travel. The priors, learned from a decade of normal years, were confidently wrong about everything.

What survived was the structure. The question was still where to open, how to sequence, how to staff. The constraints were still lead times and monitor coverage and budget. What changed was the rates and the availability of countries, and those were parameters. Within weeks the same engine was re-planning trials under new assumptions, and the value it delivered was not accuracy, since nobody was accurate that year, but the ability to compare plans coherently while the assumptions kept changing. A model that had encoded its worldview as fixed rules would have had to be rebuilt. A model that had encoded it as priors had to be updated.

That is the most philosophical thing I took from the work, and it is not about Bayes in particular. A model is useful in proportion to how honestly it separates what it knows from what it is assuming, because the assumptions are what you have to change when the world does.³

Postscript, 2026

I now work on evaluating AI agents, and I keep meeting the same structure.

An eval scenario is a site. Each one gives you a handful of runs, the runs are expensive, and the question is what to believe about a scenario that passed three of five. The naive estimate is worthless in the same way two patients in three months is worthless. The honest answer is hierarchical: this scenario starts out believing what similar scenarios believe, and earns its own estimate as data arrives.⁴

Shipping a model change is a decision under asymmetric costs. A regression that reaches users costs more than a delay, and ranking candidates by mean pass rate treats them as the same. What the decision needs is the probability of the failure that matters, under each candidate, not a leaderboard.

And the thing that makes agent evals hard is the thing that made trial planning hard: the outcome depends on paths and interactions, not on a rate. An agent’s success is a function of which tools it calls, in what order, against what state, with what failures along the way. There is no closed form. There is a simulation, which is why I spend my time building worlds for agents to act in and recording what happens when they do.

Different instrument, same question. How do you plan for a system before reality arrives, and how do you stay honest about what you were assuming when it does?

The mechanics are shrinkage: the posterior mean for each site is a precision-weighted average of the site’s own data and the population estimate, and the weight on the site’s data grows with its exposure time. What the hierarchy adds is that the population estimate is itself learned rather than asserted. ↩
Anisimov and Fedorov, “Modelling, prediction and adaptive adjustment of recruitment in multicentre trials”, Statistics in Medicine, 2007, is the standard reference for the Poisson–Gamma recruitment model. Gelman et al., Bayesian Data Analysis, is the general reference for hierarchical models. I have changed identifying details of the portfolio work; the structure and the lessons are as they were. ↩
The distinction is between a model’s parameters, which data can move, and its structure, which encodes what you believe cannot change. Every model has both. The Bayesian discipline is to make the first set as large as honesty allows and to know exactly what is in the second. ↩
The same reasoning says most eval dashboards report the wrong thing. A pass rate per scenario, computed from five runs, is a point estimate from a sparse site. What a decision needs is the posterior, pooled across scenarios that share a mechanism, with its width shown. ↩