I typed in a single enemy hero -- Necrophos -- and asked the model what I should pick.
It said Ancient Apparition.
If you play Dota, you just nodded. Ancient Apparition's ultimate prevents healing, Necrophos is a hero who wins fights by out-sustaining you, and "AA into Necro" is the kind of thing you learn in your first year and then repeat for a decade. It is the textbook answer.
My model has never read the textbook. It has never seen a hero name, an ability description, a role tag, or a patch note. It has seen integer hero IDs and one bit of outcome per match, roughly twenty-one million times.
I sat there feeling extremely clever for about ninety seconds.
Then I got suspicious, because I had just done the single most unreliable thing an engineer can do with a model: I asked it one question I already knew the answer to, and graded it myself.
The Confirmation Bias Machine
Here is the problem with eyeballing a draft recommender.
There are 127 heroes. Any ranking puts something at the top. If the top pick is a hero I recognize as a counter, I feel validated. If it's a hero I don't associate with that matchup, I have a rich vocabulary available -- "interesting", "maybe it knows something I don't", "that's a flex pick" -- and I will absolutely use it. I am not a neutral instrument. I have been playing this game for fifteen years and I have opinions about every hero in it.
Worse, "Ancient Apparition counters Necrophos" is famous precisely because it is easy to notice. The counters I could verify from memory are exactly the ones a much dumber model could also find. Testing on those is like grading a language model on the alphabet.
So the question that actually organizes this post is not "did it recommend a good hero." It's:
How do you measure whether draft advice is good, when nobody has a labelled dataset of good draft advice?
That took me a week, it produced the most useful measurement in this project so far, and the first version of it was broken in a way that flattered me.
But before any of that: I need to explain why this model could give partial-draft advice at all, because three weeks ago it couldn't.
A Model That Had Only Ever Seen Finished Drafts
Every model in Parts 1 through 6 was trained on the same thing: ten heroes, one outcome. Complete draft in, win probability out.
That is a perfectly good win predictor and a completely useless recommender, and it took me an embarrassingly long time to see why. A recommender is a thing you use during a draft. Its entire job is to answer questions about states where most of the heroes aren't picked yet. My model had never once seen such a state. Not in training, not in validation, never.
You can of course force it to answer. Feed it a draft with eight zeros in it and it will happily return a probability. That was the status quo, and I had been quietly assuming it was fine, because the model was well calibrated on the states it knew and a zero is just another input.
The obvious fix -- the one I spent a day designing before throwing it away -- is to train the model to predict the next pick. Feed it a partial draft, ask which hero comes next, learn from what really got picked. It's a clean supervised problem and there's abundant data.
It is also the wrong objective, and the reason is worth stating plainly: predicting the next pick teaches the model what players do, not what wins. Those are different, and the gap between them is the entire reason a recommender might be worth building. If everyone in Archon overrates a hero, a pick-prediction model learns to overrate it too, with high confidence, because it is genuinely what happens next. I would have built a very accurate model of the metagame's mistakes.
What I wanted was: given this partial draft, what is the probability we win?
And it turns out you already have that label. It's the match result.
The Trick That Makes the Label Free
Take a completed match. Hide some of the heroes. Keep the outcome.
That's the whole idea. If I show the model two Radiant heroes and one Dire hero from a real match and ask it to predict who won, the honest answer is the probability that Radiant wins from that state, given drafts continue the way drafts actually continue. Which is exactly the quantity a recommender needs, and exactly what the match result is a valid single sample of.
So during training, every time a match comes up, I roll dice: how many Radiant heroes are visible this time, how many Dire heroes. Zero to five each, independently. The same match gets seen in a different state of undress every epoch, and the label never changes.
Two things I changed at the same time, which is the part I nearly got wrong.
Along with the objective, I wanted a different architecture. The existing model reads heroes as ten fixed slots, which encodes something false -- that the third hero picked is a different kind of thing from the fourth. A permutation-invariant set encoder, which pools over heroes rather than concatenating slots, matches the actual structure of the problem. It should also degrade more gracefully when heroes are missing, because "a set with three things in it" is a natural input for it and "seven slots holding zero" is not.
Good argument. I believed it. And if I had shipped both changes together and measured an improvement, I would have had no idea which one did it.
So I trained all four combinations, plus a linear model as a floor. Same snapshot, same split, same 24 epochs, all scored on identical partial drafts.
I look at this chart a lot. Three things are in it.
The objective was the whole result. The two mask-trained models are the two lines at the bottom, and they are so close together that I had to dash one of them to stop it disappearing underneath the other. The largest gap between them at any state is 0.00016 -- smaller than the standard error of a log loss on this split, and at the complete draft it flips sign. My elegant architectural argument bought approximately nothing.
The architecture on its own was actively harmful. That purple line leaving the top of the chart is the set encoder trained only on complete drafts. At two heroes revealed it reaches 0.851, against 0.707 for the slot model in the same situation. Mean-pooling over a set turns out to be more sensitive to a change in set size than zero-padded slots are, which is the exact opposite of the graceful-degradation argument I made for it.
The status-quo model was worse than nothing. The red line sits above the dotted base-rate line for every state up to and including three-versus-two. A model that has never seen a partial draft, asked about a partial draft, gives answers worse than knowing only that Radiant wins 53.15% of the time.
I want to sit on that last one, because it is the thing I would have shipped. I had a good model. Its aggregate numbers were the best in the project. Asked the question the product is actually built around, it was worse than a constant.
There's a small joke in the far-left column too. At zero heroes revealed the set encoder scores 0.69116, which is the base rate, and the slot model scores 0.69315, which is exactly ln 2. Given nothing at all, the slot model returns a hardcoded logit of zero -- it doesn't even know which side wins more often. The set encoder does. It's the one structural win in the whole comparison and it is worth precisely nothing, because nobody asks a recommender for advice on an empty draft.
So: keep the objective, throw away the architecture, and feel grateful I ran the boring combination that let me tell them apart.
The Answer Key
Which brings me back to Ancient Apparition, and to grading.
Here's the observation the rest of this post rests on. I don't have labelled draft advice. But I have twenty-one million matches, and in a great many of them, somebody already ran the experiment.
Want to know whether Ancient Apparition is a good answer to Necrophos? Find every match where one team had an Ancient Apparition and the other had a Necrophos, and see who won. That's not a model's opinion. That's what happened.
So I built what I've been calling the oracle: for a given draft state, rank every legal candidate hero by the model's score, then look up the real win rate of matches that actually reached that combination. Two things fall out. How well does the model's ordering correlate with reality, and -- the number I care about more -- if you take the model's top five suggestions, what do those actually win?
That second one is a product metric. It has units. It's percentage points of win rate, and I can put reference marks around it: what you get picking at random, what you get by ignoring the model entirely and taking the highest-win-rate hero available, and what you'd get if you chose perfectly.
The power is better than I expected, because of a counting mistake I'd made in the design doc. I'd worked out how many matches support each pair and concluded the whole thing was underpowered. I was off by a factor of twenty: each match contributes ten hero pairs per team, and there are two teams. Cross-team cells -- our hero against their hero, which is what countering means -- are better still, at 42,193 matches each. That's a standard error small enough to argue with.
Here's the first real reading, on one enemy hero:
| win rate of the top 5 picks | |
|---|---|
| pick at random | 0.5000 |
| ignore the model, take the strongest hero | 0.5408 |
| the status-quo model | 0.5339 |
| the mask-trained model | 0.5497 |
| best possible choice | 0.5538 |
The trivial baseline in bold is the one that matters. "Rank heroes by their own overall win rate and ignore the draft entirely" is the dumbest possible recommender, it takes four lines of SQL, and it captures most of what's available. Any model has to beat that, not random.
The mask-trained model beats it by 0.890 percentage points, paired across contexts, t = +11.91.
The status quo loses to it by 0.690 points. Masking a complete-draft model at inference produces advice worse than ignoring the model and picking the strongest hero available -- which is, again, what I had.
The Metric That Loved a Model With No Opinions
The first version of this measurement was wrong, and it was wrong in the most dangerous direction: it told me things were going well.
The problem is that most of what makes a hero good in a given spot is just... the hero being good. Wraith King wins more games than Meepo in almost every context. If I correlate my model's ranking against real win rates, I mostly measure "does it know Wraith King is strong," which is not the interesting question. The interesting question is whether it knows anything about interaction -- whether this hero is better than usual specifically here.
My first instinct was to subtract each hero's overall win rate from both sides and correlate the remainders. Take out the additive part, see what's left.
This is nonsense, and it took a control to show me it was nonsense. Subtracting the same term from both sides of a correlation doesn't remove that term's influence; it manufactures agreement between the residuals. I found this out by running the metric against a model that returns the same number for every hero. A constant predictor scored +0.0343 -- higher than the actual linear model I was comparing against.
The fix is a partial rank correlation, which conditions on hero strength rather than subtracting it. Under the corrected metric a constant predictor returns undefined, as it should, and a uniform random one scores -0.0024.
The real lesson isn't about rank correlation. It's that I only caught this because I ran the metric against models that know nothing. Those controls now travel with every oracle result automatically, in the same JSON payload, because the version of me who is excited about a number is not the version of me who should be deciding whether to check it.
On the corrected metric, the mask-trained model scores +0.6545 on countering. The linear model -- which cannot represent interaction at all, being a sum of per-hero weights -- scores +0.0265. That gap is the first direct evidence in this project that the network learned something about pairs of heroes rather than a very good tier list.
One of These Results Was a Lie
Everything above is measured on the training split. The empirical win rates and the model's weights come from the same matches, which means it demonstrates that the model recovered structure it was fitted on. Necessary. Not sufficient.
So I recounted every win rate on the validation split -- 2.67 million matches the model has never seen -- and asked the same questions again. Both columns below take the baseline's hero win rates from the training split, because building the baseline out of the same matches you score it against flatters it, and I would rather the trivial baseline be too strong than too weak.
| question | training split | held out | t | |
|---|---|---|---|---|
| countering one enemy | +0.894 pts | +0.824 pts | +10.10 | holds |
| synergy with one teammate | +0.244 pts | -0.165 pts | -3.24 | gone |
Countering survives almost intact. An 8% attenuation across a split boundary, still ten standard errors from zero. On matches it has never seen, the model gives better counter-picks than the trivial baseline.
The synergy result doesn't just weaken. It reverses sign. On unseen data, the model's teammate suggestions are worse than ranking candidates by their own win rate.
That one hurt, and then it made sense. With a single teammate locked, the gap between the trivial baseline and the best answer my answer key can even express is only 0.50 points, and my model had appeared to take half of it. In hindsight that was too good, and I should have flinched at it rather than screenshotting it. There is very little real synergy signal available at one hero, so a model with enough capacity fits the noise in the pair counts, and the noise doesn't transfer.
Countering has 1.30 points of headroom -- 2.6 times as much -- and 5,165 matches per cell on validation against 3,994 for within-team pairs. More signal, more data, and it holds.
So the honest description of what I have built is narrower than what I set out to build. It is not a draft advisor. It is a counter-picker. It is good at reacting to the enemy draft and it has not demonstrated that it can help you build a team that works together.
I would rather know that than not.
What Necrophos Actually Does to It
With the measurement in place, I finally went back and looked at the recommendation that started all this -- not to grade it by vibes, but with the empirical lift printed next to every hero.
One enemy pick, and the list rearranges itself. Ancient Apparition goes from 71st to 1st. Grimstroke 62nd to 3rd. Skywrath Mage 43rd to 4th. Meanwhile Wraith King -- the highest win rate hero in the game at 0.5483 -- falls from 1st to 39th, and Spectre from 2nd to 29th.
This is not hero strength wearing a disguise. Ancient Apparition's overall win rate is 0.5113 and Skywrath Mage's is 0.4993, at or below average, and against Necrophos specifically their real rates lift by 2.35 and 1.26 points. Wraith King and Spectre are the two strongest heroes in the game and their rates against Necrophos fall by 6.44 and 5.62 points. The model demotes them correctly.
And there's an asymmetry in that picture I didn't expect, which is now one of my favourite findings.
Of the nine heroes the model demotes hardest when Necrophos appears, nine out of nine have a negative real lift against Necrophos, averaging -5.15 points. Of the nine it promotes hardest, seven of nine are positive -- but the average is only +0.47 points, and two are outright wrong (Outworld Destroyer at -2.7, Viper at -0.6).
The model is dramatically better at knowing what not to pick than at knowing what to pick.
Which fits the shape of the data. Heroes that lose to Necrophos lose to him consistently and by a lot. The upside of a genuine counter is smaller and more diffuse. And for an actual product this is the more useful half anyway -- a recommender that reliably removes the traps is worth having even when its top suggestion is only marginally better than its third.
Best Epoch 48, of 48
One piece of housekeeping first, because it decides which numbers can sit in a table with which.
Partway through all this I took a fresh snapshot -- 27.7 million matches where the old one had 26.7 -- because my collector had started capturing something I wanted and could not get any other way. I'll come back to what. Retiring a snapshot costs you exact comparability with everything measured on the old one, which is a real price, so I retrained the champion on the new cohort before trusting any of it. Every measure reproduced to within 0.0003.
Everything from here down is on the newer cohort. Everything above it is not, and I have kept them out of the same table.
Now. In Part 6 I spent a week making a hyperparameter search affordable, and closed by noticing that every run I had ever done picked its final epoch as its best. Best epoch: 8, of 8. I called that the tell -- a model that improves right up to the moment you stop it is telling you that you are the binding constraint.
So I ran the search. Nine trials queued, spanning embedding widths from 64 to 256, two to four layers, dropout from zero to 0.2, a tenfold range of learning rates and a two-hundredfold range of weight decay.
I killed it after five, because five was enough. One of those five collapsed into a constant predictor, which an automatic abort caught after a single epoch and twelve minutes rather than the two hours it would otherwise have burned. The other four landed within 0.00015 of each other on partial log loss. The standard error of that metric on this split is 0.00033. My entire search fits inside half a standard error.
And every one of them selected epoch 23 of 24. Still improving. Same tell.
So I doubled the schedule and ran it again. Best epoch: 48, of 48. Still improving, exactly as Part 6 taught me to expect.
Here is what those extra twenty-four epochs actually bought, broken out by how much of the draft the model can see:
| heroes known | 24 epochs | 48 epochs | gain |
|---|---|---|---|
| 1 | 0.690033 | 0.690017 | -0.000016 |
| 2 | 0.688653 | 0.688633 | -0.000020 |
| 4 | 0.685228 | 0.685140 | -0.000088 |
| 6 | 0.680962 | 0.680806 | -0.000156 |
| 8 | 0.675172 | 0.674966 | -0.000206 |
| 10 | 0.667850 | 0.667491 | -0.000359 |
Every bit of it is at the right-hand end.
At one hero known -- which is precisely where a recommender gets asked its first question, and precisely where the countering measurement is taken -- the gain from doubling the training budget is 0.000016. That's five percent of one standard error. Nothing. And sure enough, none of the recommendation metrics moved: the countering margin went from +0.890 points to +0.873, comfortably inside its own error bar, and the interaction score went down.
I also ran a model with twice the width and 50% more depth, at the same doubled schedule, and let it converge properly this time. It matched the small model's partial log loss to six decimal places, at 2.3x the compute, with the worst countering margin of the batch.
So Part 6's lesson survives, but with a caveat I did not see coming. The model was still learning. It just wasn't learning anything the product uses. It was getting better and better at predicting the outcome of finished games -- and a finished game is the one draft state a draft recommender never has to think about.
I bought a search and it told me my defaults were fine. That's the cheque Part 6 wrote, and I'm cashing it honestly: the interesting variable was never in the search space.
Forty-Four Percent
Which raised the obvious question. If it isn't epochs and it isn't parameters, what is it?
Which is where the new snapshot comes back. The thing I wanted badly enough to retire a cohort for is draft traces: for a subset of matches, my collector captures the draft as it happens, the actual sequence of picks in order. The old snapshot had 96 of them. The new one has 5,121.
I had been thinking of those as material for a future post about pick order. Instead they answered a question I hadn't thought to ask. Expand each trace into its eleven prefix states, from empty draft to complete, and you get 56,331 draft states that a real game actually passed through.
I'd been generating training states by rolling two independent dice: zero to five heroes visible on each team, uniformly. Thirty-six states, all equally likely. It seemed like the natural choice and I never questioned it.
Real drafts reach twenty of the thirty-six states. The other sixteen are not rare. They are hard zeros -- not one occurrence across all 5,121 captured drafts.
Of course they are. Ranked All Pick alternates picks between the two teams, so they are never more than a couple of heroes apart. A state where I have five heroes and you have none is not unlikely; it is unreachable, and I have been diligently training my model to handle it since the day I wrote the masking function.
Forty-four percent of every gradient I have computed has been optimizing draft states that cannot occur.
And it's worse than the headline, in a quiet way. The states the oracle actually evaluates -- one enemy hero and nothing else, one teammate and nothing else -- are among the most common real states, at around 4.5% each. The uniform mask gives them 2.8%. The states I care most about are the ones I've been underweighting.
I want to be careful about what this is and isn't. It is a measured defect in my training distribution with a clear mechanism. It is not yet a result. I have not trained the reweighted model, and it is entirely possible that a network with capacity to spare -- and I have now proven mine has capacity to spare -- simply doesn't care, in which case those impossible states were harmless noise and I'll say so.
But it is the first lever I've found in a month that isn't about spending more compute. Every other knob I've turned lately makes the model better at something it already does. This one changes what it's asked to be good at.
Takeaway
The engineering lesson here is about the gap between a metric and a purpose, and how quietly they come apart.
I had a loss function that was genuinely, honestly going down. Every run improved it. Doubling the budget improved it. Nothing about that was faked or mismeasured -- the number really was getting better, monotonically, for a full day of GPU time.
And it was getting better almost entirely in the one regime my product will never see. If I had kept optimizing the number I was watching, I would have kept congratulating myself, indefinitely, on a model that was becoming a superb historian of finished games.
The thing that caught it wasn't a better loss function. It was building a separate measurement, in different units, pointed at the thing I actually wanted -- percentage points of win rate on advice, with a trivial baseline sitting next to it refusing to be impressed. It cost me a week. It found a sign-flipped result I would otherwise have published, and it's the reason I know that "still improving" and "still improving at anything useful" are different sentences.
Build the answer key before you need it. You will not enjoy what it says, and that's the entire point.
Next time: I retrain on the twenty states that exist. Either 44% of my compute has been going somewhere useless, or it's been a free regularizer this whole time, and I genuinely don't know which.
One more thing from the trace data that I'm not ready to do anything with.
Those 5,121 traces don't just tell me which states occur. They carry the order -- which hero was picked first, second, ninth. That is real information my model has never had access to, and there are good reasons to think it matters. A hero taken in the first phase means something different from the same hero taken last, because the first one was chosen blind and the last one was chosen as an answer.
The catch is arithmetic. Of those 5,121 traces, thirteen fall in my validation split. Thirteen. I could train something order-aware tomorrow and I would have no way on earth to tell whether it worked.
So the collector keeps running, and I wait. Which is a familiar feeling in this project -- Part 5 was entirely about a month I spent making a data collector fast, and here I am again, blocked on the same resource, for a completely different reason.
Twenty-seven million matches, and I need a few thousand more of a very specific kind.