Team Builder: the full method
How the Team Builder turns a roster into a projected chance of clearing 80/100, how it was tested on films it had never seen, and what it cannot know.
In plain English
We start from how often films of this genre have cleared 80/100 in recent years, which differs enormously between genres — that starting point is the single largest influence on the result. Then we look at each person on the roster and ask a narrower question than “what is their batting average”: did their films beat the average film of the years they were working? Take two directors who each cleared the bar in one film out of three: in a decade when one film in eight was a hit that is a remarkable record, and in a decade when nearly half were, it is a poor one. They are not given the same credit.
What happened when we tested it
The model is built on an earlier block of years and then graded on a later, separate one, scored once with the final settings. These are those results.
The bar we set before measuring
Four conditions were written down before the holdout was scored: it has to sort films better than chance by a clear margin, its probabilities have to beat a constant forecast, its top tenth has to contain hits at least twice as often as an average film, and its stated chances have to match observed outcomes closely. All four must pass. If any fails, the Team Builder shows no projection at all — not a softened one.
Where these figures come from
The model and the results above live in one committed file, so the accuracy shown on the site cannot drift away from the model that produced it — they arrive together or not at all. The monthly data refresh refits the model on the new data and rewrites that file, and a check in continuous integration compares every figure this page displays against a fresh measurement, failing the build if any of them no longer describes the data. That is why none of these numbers is typed into the page.
Related
How these numbers work: Methodology