In Part 1, I left you with a model that had successfully trained for one full epoch and learned a single deep truth about Dota 2: Radiant wins about 53% of the time. I described this with the affectionate exasperation it deserved and promised a sequel.
What I did not mention -- because it would have ruined the cliffhanger -- is that I let it keep training. Forty-three epochs. Early stopping kicked in. The checkpoint was saved. Everything looked extremely professional and legitimate. I poured a coffee, opened the eval script, and waited for the part where my Transformer announces that it has, in fact, mastered the deep semantics of Dota.
Test Accuracy: 54.77%
Predicted Radiant WR: 70.0% (actual: 53.2%)
Calibration: Poor
54.77%. The baseline is 53.2%. After 1.1 million parameters, 4.17 million training examples, multi-head attention, position embeddings, gradient clipping, Pre-LN, and a NaN incident heroic enough to deserve its own paragraph in the previous post, my model had managed to climb one and a half percentage points above "always guess Radiant."
This is the part of an ML project where you sit very still for a minute and consider your life choices.
Part 1: The Baselines Stab Me In The Heart
Before doing anything dramatic, I needed to figure out whether the problem was the model or the data. The standard move here is to throw a bunch of much simpler baselines at the same dataset and see whether any of them do better. If a simple baseline does well, your fancy model has a bug. If no baseline does well, the problem is the data.
I started with the dumbest possible baseline: a hero win rate lookup. For each hero, compute its average win rate across the training set. To predict a match, average the five Radiant heroes' win rates and the five Dire heroes' win rates and pick whichever side is higher.
Result: 53.90%. Six tenths of a point above "Radiant always wins." Heroes are, on average, balanced enough that picking a single hero barely moves the needle.
Then a real baseline: a tiny MLP. Sum the hero embeddings on each team, concatenate the two team vectors, push them through a couple of dense layers. No attention. No transformers. Permutation-invariant. Thirty-seven thousand parameters total. About 30x smaller than the Transformer.
Result: 54.88%.
The toy MLP matched my Transformer. With 30x fewer parameters. Without an attention mechanism. All those carefully designed cross-team attention paths I waxed poetic about in Part 1 -- the synergies, the counters, the "single Transformer over all 10 slots handles this naturally" line I was particularly pleased with -- contributed exactly zero measurable benefit over summing some embeddings and running them through a linear layer.
Embarrassing? Yes. Useful? Also yes, because now I had information.
While I was at it, I tested one more thing. Part 1 mentioned the 10x data augmentation trick: each complete match becomes ten partial-draft training examples (1 pick, 2 picks, ..., 10 picks). It seemed obviously good. More data, broader draft state coverage, model learns to handle partial drafts.
I retrained both models on complete drafts only -- one example per match instead of ten:
- Simple MLP: 55.74% (+0.86 pp)
- Transformer: 55.90% (+1.13 pp)
Training on one tenth of the data made the model better. The augmentation, the thing I had spent a chunk of Part 1 admiring my own cleverness about, was actively hurting performance.
(In hindsight: the augmentation generated partial drafts in a fixed alternating pick order. The model wasn't learning "hero X synergizes with hero Y." It was learning "the hero in slot 0 matters more than the hero in slot 7," which is true of my augmentation but not of real drafts. Garbage augmentation in, garbage signal out.)
So my baselines had stabbed me twice in five minutes:
- The Transformer was no better than a 37k-parameter MLP.
- The augmentation that I was very proud of was making things actively worse.
Fine. So the architecture wasn't the problem. The problem was somewhere in the data.
Part 2: The Hypothesis That Felt Obvious
Here is the thing every ML practitioner does when their model fails to learn from a dataset, and here is the thing I did: I started suspecting the data quality. Pub matches are noisy. Player skill matters more than draft. Anyone who has watched enough Herald reviews (shoutout Jenkins) should know that five melee versus Earthshaker is going to be a bad time. So you'd expect, intuitively, that high-rank games are where draft actually matters, because high-rank players are competent enough to actually draft.
The hypothesis went something like this:
In low-rank games, players don't understand drafts. They pick five carries. They first-pick Brood into Earthshaker. They make confused emotional decisions in the queue. The win/loss signal in those games is dominated by execution, by skill gaps, by the simple fact that the higher-MMR player makes fewer mistakes per minute. Draft composition barely matters.
In high-rank games, players are real. They draft on purpose. They understand counters. They build cohesive line-ups. The signal in draft composition matters the most when you are at the top of the ladder.
I was extremely confident about this. I had a Phase 2 plan. I was going to filter the dataset down to Divine and Immortal matches only, train the model on those, and watch the accuracy jump.
OpenDota provides an avg_rank_tier field on every match. I filtered to tier 70 and above (Divine+). The dataset shrank from 417,000 matches to 32,080. I retrained the Transformer with the same hyperparameters. I watched the loss curve descend, the same way I'd watched it dozens of times before. The model finished training. I opened the eval script.
Test Accuracy: 53.77%
I sat with that number for about a minute.
53.77 is worse than the 55.90 the unfiltered model got. The hypothesis hadn't just failed. It had failed in the opposite direction. Filtering to the matches where I was certain the signal would be strongest had made the model 2.13 points worse.
I did the only reasonable thing, which was to assume I'd messed up the filter. I checked the SQL. The filter was correct. I checked the labels. The labels were correct. I retrained from a different seed. Same result.
OK. Maybe the hypothesis isn't right.
Part 3: The Ladder Climbs Down
If high-rank wasn't the answer, what was? I wasn't going to guess again. I wrote a script to train one separate model per rank bracket -- Herald, Guardian, Crusader, Archon, Legend, Ancient, Divine -- with identical architecture and hyperparameters. Same training time, same validation split, only the rank filter changed.
This took a day to run. When it finished, I opened the chart.
The shape was so unambiguous I actually chuckled at my desk.
The model is most predictive at the bottom of the ladder. Herald games hit nearly 56% accuracy. Guardian, Crusader, Archon, Legend, Ancient -- the line descends almost monotonically. By the time we hit Divine, accuracy has dropped to 53.4%, which is essentially noise around the Radiant-win baseline.
My hypothesis was not slightly off. It was inverted. The skill axis points the opposite direction from what I'd assumed. Low-rank games carry the strongest draft signal. High-rank games carry almost none.
Part 4: Why (In Retrospect, Obviously)
Once I saw the chart I sat with it for a while, and the explanation rearranged itself in my head into something that should have been obvious from the start.
I had been thinking about this as a question of signal-to-noise: low-rank games are noisier, high-rank games are cleaner, so the cleaner data should yield the better model. But that framing assumes the signal exists in equal strength in both populations and you're just trying to filter the noise off it.
That's not what's happening. The truth is much simpler:
High-rank players don't make bad drafts in the first place.
Every Dota player has lived through some version of this scene. A Herald support player locks in Legion Commander, types "i jungle 4" in chat, marches off into the trees, and emerges thirty minutes later with a Phase Boots and zero kills, while their actual safelane carry has been getting stomped 1v2 since the second minute. This is the kind of pick that doesn't get to make it to the late game and then turn it around. It loses the game in the first picking phase. It is a real, recognizable, punishable mistake, and it shows up in the dataset over and over.
It is also exactly the kind of pick the model can learn from. If "no second support, plus a position-4 Legion Commander, plus an offlane that should have been a melee initiator" reliably correlates with losing, the model can fit a function around that. Replicate the pattern across hundreds of thousands of matches in the lower brackets, and you get an actual signal.
In Divine, that draft does not exist. Divine players might experiment with unconventional picks -- a position-1 IO (shoutout Ana), a pos 4 Naga -- but those are intentional unconventional picks, executed by people who know what they're trying to do, often successfully enough to keep the win rate roughly even. The visibly broken drafts the model would need to learn from are not in the high-rank data. The reason high-rank models can't predict draft outcomes is not that the players are too good at overcoming bad drafts. It's that there are no bad drafts in the dataset. Every team comp on every side, in every game, is at least passably reasonable.
So:
The signal in draft data isn't "what makes a good draft good." It's "what makes a bad draft bad." And bad drafts only exist where the players making them don't yet know any better.
The model isn't a Dota strategy oracle. It's a Herald-Archon dictionary of recognizable compositional disasters: pos-4 Legion Commander junglers, five-melee-into-Earthshaker, first-picked Brood when the enemy is about to first-pick Shaker. As skill rises, those entries gradually disappear from the dataset, and by Divine the entire dictionary is empty.
This was, in the language of my own past blog posts, a problem of who, not what. I'd spent two weeks tuning the what -- the architecture, the augmentation, the hyperparameters -- and the answer was a who. The wrong players were in my dataset, in the very specific sense that they didn't make the mistakes the model needed to see.
There's a related, slightly funnier thing the rank-by-rank charts told me. The high-rank models were all better calibrated than the low-rank models. The Divine model's predicted Radiant win rate was within 0.1 percentage points of actual. The Guardian model overpredicted Radiant by nearly 19 points. The high-rank models had given up on extracting a draft signal entirely and were correctly learning that at this skill level, every game looks the same to me, so I'll just predict 50/50. That is in fact the right thing to predict, given the data they had. It just is not useful to anyone.
Part 5: The Filter
I changed one line of SQL.
WHERE avg_rank_tier < 50 -- Herald, Guardian, Crusader, Archon
The dataset shrank from 417k matches to about 200k. I retrained.
Test Accuracy: 56.27%
That's a 0.37-point bump over the unfiltered model and a 2.5-point bump over the high-rank-only model. More importantly, it was the first time the model had given me a number I felt I could trust -- a number that came from a clean signal in clean data, instead of a confused average over a population the model couldn't make sense of.
That filter never came off. Every model I trained from that point on, every experiment in the months that followed, every hyperparameter sweep, every breakthrough config -- all of it ran on Herald-through-Archon games. rank < 50 is the most important line of code in the entire project, and it survived from this single afternoon's investigation all the way through to production.
The model isn't going to learn anything about Immortal-level Dota. That's fine. I am not an Immortal. My friends are not Immortals. When we queue up for a pub game, somebody on the team is going to type "pos 5 pudge" and lock in Pudge -- and a model trained on people exactly like us is, it turns out, the model I actually want.
Takeaway
I keep coming back to one detail of this story: I had a strong intuition. I wrote a six-phase debugging plan around it. I was prepared to file the rank filter under "obvious win, ship it, move on." And the very first experiment I ran dismantled the entire premise.
The model doesn't care about your intuitions. It doesn't care that you've played Dota for twelve years, that you "know" draft matters at high ranks, that you have read the patch notes and watched the Internationals. It just learns patterns from the data you give it, and if you give it the wrong data because of a confidently-held belief about how the game works, it will silently fail to learn, and only the test set will tell you.
If there's a slogan I'd take away from this, it's that in any ML project, the boundary you draw around your dataset is itself a hypothesis. Filtering matches by rank wasn't preprocessing. It was a research question dressed up as a SQL clause: which players are still making the kinds of mistakes I want my model to learn from? Most projects I've worked on, the dataset was just there, taken as a given. This one was the first time I had to admit that deciding what data to train on was the most consequential modeling decision of the entire project.
Also: never trust your data augmentation. If you wrote it on a Saturday and it 10x'd your dataset, it is silently lying to your model. That one's free.
What's Next
I now had a model that reliably hit 56.27%, a hard-won 3 points above the dumb baseline. It was trained on the right data, after the right architecture, with the right augmentation strategy. I was, in principle, ready to start improving it.
Which meant tweaking hyperparameters. One at a time. Twenty-five minutes per run. Writing each result down in a notepad and bumping a parameter by 0.0005 and submitting another.
I gave it a weekend and realized it was awful.
That's Part 3.