Last time I promised to retrain on the twenty draft states that actually exist, and said I didn't know whether the other sixteen had been wasting 44% of my compute or quietly regularizing the model.
Neither. They were doing nothing at all, in a very specific and annoying way.
I ran it three times. Restrict training to reachable states: no change where it matters. Weight those states by how often they really occur: no change where it matters. Then the version I was actually confident about -- pour 80% of the training signal into early-draft states, up from 38.9%, because early drafts are the only thing a draft recommender is ever asked about.
Here is what a doubling of the training budget on the states I care about bought me:
| heroes known | before | after | change |
|---|---|---|---|
| 1 | 0.690016 | 0.690001 | -0.000015 |
| 2 | 0.688626 | 0.688593 | -0.000033 |
| 3 | 0.687052 | 0.687073 | +0.000021 |
| 4 | 0.685144 | 0.685216 | +0.000072 |
Two of them got worse. I had written the pass/fail threshold down before the run -- a real improvement had to clear 0.00003 -- precisely so I couldn't squint at a table like this afterwards and talk myself into something. Good decision. Deeply unsatisfying to be right about.
That's six consecutive experiments now: a new architecture, more epochs, more parameters, a nine-trial hyperparameter search, and three different training distributions. Every single one improved the model at the complete draft and did nothing at the early draft. Six for six. You could set a clock by it.
At some point a run of results that consistent stops being a series of disappointments and starts being a message.
The Part Where I Stop Typing
There's a specific feeling when the next experiment on your list is one you already know the answer to. I had two more queued. I could see how they'd go.
So I closed the terminal and did the thing I'd been avoiding, which was to write down everything I had done in this project -- every claim, every number, every "established" -- in one document, and then read it as though someone else had written it and I was being paid to find the flaw.
I want to be honest about how that felt, because "I critically examined my methodology" sounds like a thing you say in a retro and not like a thing you do on a Tuesday evening with a growing sense of unease. It's not a fun exercise. You are specifically hunting for the moment where past-you was pleased about something, because that's where the mistakes live. Every reassuring sentence is a suspect.
I found the first one on the third day. It was on page two, in a table I had made myself, and I had looked at it maybe forty times.
Beaten by a GROUP BY
Here is the table. This is the core question of the whole project: the enemy has one hero, what should you pick? For every candidate, I look up the real win rate of actual matches where our team had that hero and theirs had the enemy hero, and I take the average of my model's top five.
| real win rate of the top 5 | |
|---|---|
| pick the strongest heroes overall, ignore the draft | 0.54078 |
| my neural network | 0.54957 |
| a lookup table | 0.55283 |
That third row.
The lookup table is not a model. It has no parameters. It does not train. It counts how often each hero pair won, sorts, and takes the top five. To make sure it isn't cheating by scoring itself on its own noise, it computes the ranking from half the matches and gets scored on the other half -- so it's a lookup table working with one hand tied behind its back.
It beats my Transformer by 0.33 percentage points.
I have 22 million matches, a GPU, six weeks of experiments, and a model with attention in it. I was comprehensively outperformed by a GROUP BY.
And the number wasn't new. I had computed it two weeks earlier and put it in a table labelled ceiling. I'd been treating it as the theoretical maximum, the thing to measure my distance from -- "we capture 73% of what's achievable!" -- when it was sitting right there in the same column as my model, winning. I had been quoting my own defeat as a progress metric.
The reason I didn't see it is embarrassing and completely mundane: I'd put it in a row called "ceiling" and ceilings are things you don't reach. The label did my thinking for me.
But it does resolve something. I'd been slowly convincing myself that the early draft just doesn't contain much signal -- that one or two heroes can't tell you enough, and I was near the limit of what's knowable. That story is dead. A dumb estimator cannot find signal that isn't there. If a lookup table beats my model, the signal exists and my model isn't getting it. That's not a wall. That's a bug with better PR.
Three More, In Increasing Order of How Much They Hurt
Once the first hole opens, the rest arrive quickly.
"Skill isn't the explanation." I'd been worried that my hero-pair effects were really just measuring that better players pick better combos. So I split the data by rank bracket and checked whether the pair effects agreed between the bottom brackets and the top ones. They correlated at +0.55, on disjoint populations. I wrote down that skill was ruled out and moved on, feeling clever.
It rules out bracket-average skill. It says nothing about the confound that actually matters, which is that a hero's win rate in a matchup is measured on the players who chose that hero -- who are disproportionately specialists in it. Ancient Apparition looks good against Necrophos partly because the people picking Ancient Apparition into a Necrophos are Ancient Apparition players.
Specialists exist in every bracket. So the confound reproduces across brackets exactly as cleanly as a real effect does. My test was consistent with the thing it was supposed to rule out. And it biases in the worst possible direction -- it inflates precisely the niche counter-picks my model looks smartest recommending.
I went to check how big it is, and found that my snapshot contains no player identifiers at all. I can't measure it. I can't correct for it. The best I can do is write it in the limitations and stop making the claim.
The draft doesn't alternate. Small, but it stung. I'd written -- in the research notes, in the code comments, and in the last blog post -- that drafts alternate picks between the teams. It's how I explained which draft states are reachable. I finally checked it against the 5,121 real drafts I'd captured.
Picks come in blocks of two. There are 72 distinct team orderings and the most common one accounts for 3.2% of drafts. I had asserted a mechanism about a game I've played for fifteen years, in a post, twice, without once running the two-line query against data sitting on my own disk.
And then the one that reorganized everything.
The Question I Never Asked
Every evaluation in this project measures how good the advice is at some specific draft state. Countering: they have one hero, you have none. Synergy: you have one or two, they have none.
I picked those states about two months ago. I picked them because they're the states with the most matches behind them -- the ones where the measurement is statistically strongest.
It had genuinely never occurred to me to ask whether they're states anyone is ever in.
So I asked. My product is a live draft advisor: you're mid-draft, you enter what's on the board, it tells you what to pick. Which means it gets queried at exactly one kind of moment -- the instant before someone picks, from the point of view of whoever's picking. I have 5,121 real drafts. Expanding them gives 51,210 such moments. I counted where they land.
Fourteen. Of thirty-six.
A live advisor is never asked about twenty-two of those states. Half its queries have four or fewer heroes on the board. And it is never, ever asked what to pick when your own team is already full, which is obvious the moment you say it out loud and which no training distribution I have ever used reflected.
Look at the red outlines on the right-hand panel. Those are two of my three evaluation questions. They sit on cells with zero query mass. For two months I have been carefully measuring the quality of advice in situations that cannot occur.
The green one is countering. That one's real -- 6.1% of queries. It is also, and I don't think this is a coincidence, the only one of the three that ever survived a held-out test.
And the training distributions? All wrong, including all three of the ones I "fixed" this month. The uniform mask spends 44% of its effort on states that never happen. The empirical version I was so pleased with spends 9.1% of training on the complete draft -- a state where there is nothing left to recommend and no question to answer. My clever 80% early-draft weighting spread itself over 35 states, of which 21 are never queried.
Six failed experiments, and every one of them was tuning how to allocate effort across a set of states that was wrong to begin with.
What I Actually Got Out of This
A week of hostile self-review produced no new model and no better number. It produced something more useful, which is a project that can now be wrong in a detectable way.
A real competitor instead of a strawman. I'd been comparing against "pick the strongest heroes and ignore the draft," which is a baseline chosen to be beaten. The lookup table is the honest opponent, and it's currently ahead.
A thesis that can fail. The lookup table has 43,746 matches for every one-hero matchup and about 759 for a three-hero combination. It's excellent where it has data and it turns into noise where it doesn't -- which is most of a real draft. My whole argument for building a model is that it should degrade more gracefully. That's testable: race them both down the draft, one hero each, then two, then three, then four, and find the crossover. If the model overtakes the lookup partway down, that crossing point is the product. If it never does, I don't have a product yet and I should say so rather than tune another hyperparameter.
Fourteen squares. I know which states to train on, which states to measure, and which states to ignore, for the first time since I started.
Takeaway
The lesson isn't "test your assumptions," which is the kind of thing that's true, free, and changes nobody's behavior.
It's more specific and more uncomfortable. For four months I optimized a model, and for zero minutes did I define the decision problem it was solving. Those feel like the same activity from the inside. Both involve reading papers, writing code, and looking at numbers going down. Only one of them requires you to answer an embarrassingly simple question -- when will someone actually use this, and what will be on the screen when they do -- and that question has no gradient, no dashboard, and nothing to tune.
So I skipped it, and every technically-correct thing I built afterwards inherited the omission. The evaluation measured impossible situations. The training distribution spent 9% of its effort on a state with no question in it. Six experiments carefully optimized the allocation of a budget across the wrong set.
The model was never the bottleneck. I was, in the most boring way available: I never wrote down what I was building.
Next time: the race. Lookup table against neural network, all the way down the draft. I have a guess about where they cross, and given the last six weeks I'd advise against putting money on it.
One small thing from the audit that I can't stop thinking about.
The tool is most valuable in the week after a patch drops. That's when the meta is unknown, when everyone's tier list is stale, and when a draft assistant would genuinely be worth having.
It's also, necessarily, the week when I have the least data about the new patch.
I don't have an answer. I just think it's funny, in the specific way that things are funny when they're structurally rigged against you and you didn't notice for four months.