Part 5 ended with a loose end I'd tied badly. Twenty-six million matches had just told me that Bristleback loses eight percentage points of win rate between Herald and Divine while Enigma gains seven, and I signed off by saying rank wasn't a filter, it was a variable, and that was Part 6's problem.
So let me deal with it first, because dealing with it is what sent me down the road this post is actually about.
The Restriction I'd Been Honoring for Four Months
Back in Stage 2 -- this is early March, two rewrites of the data pipeline ago -- I ran an experiment that said low-rank matches were much more predictable than high-rank ones. 56.27% on the low brackets against 53.77% on Divine and above. Two and a half points.
That felt right. Better players presumably do more with the same heroes; the draft explains less of the outcome; the model should struggle up there. I restricted training to Herald-through-Archon and got on with my life. Every model I have trained since has honored that restriction.
Now I have a snapshot with a fingerprint, a recorded split, one game mode, and 21.4 million training matches, so I asked the same question properly. Fit inside each bracket, evaluate inside each bracket, one population, one split:
| Bracket | Test matches | Fitted within bracket |
|---|---|---|
| Herald | 107,264 | 56.82% |
| Guardian | 332,028 | 57.17% |
| Crusader | 481,825 | 56.62% |
| Archon | 554,941 | 56.54% |
| Legend | 473,457 | 56.23% |
| Ancient | 300,890 | 55.94% |
| Divine | 241,705 | 55.99% |
The direction survived. The magnitude did not, and neither did most of the specifics.
The real spread is 1.23 points, not 2.5. The peak is Guardian, not Herald. And Divine -- the bracket I'd written off as intrinsically noisy -- is not the worst. It's level with Ancient, and slightly ahead of it.
The original experiment had 38,850 Herald matches against 89,322 Crusader, mixed game modes, and no recorded split. It wasn't wrong so much as it was three claims stacked on a foundation that could support about one of them.
Here's the part that actually stung. The old finding said high ranks are unpredictable. The new finding says something almost opposite: high ranks are perfectly predictable if you price heroes the way that bracket prices them. Fitting inside Divine rather than using the pooled model is worth a full point of accuracy there, against under 0.11 points in the middle brackets. The pooled model isn't confused by Divine. It's confidently applying Archon's opinions to it, because Archon is where most of the data lives.
I hadn't found a limit of the data. I'd found a limit of a model that wasn't allowed to know who was playing.
The Number I Got to Predict
So I gave the model a rank embedding and trained it twice, identical in every other respect.
| Model | Accuracy | Log loss |
|---|---|---|
| Signed bag-of-heroes | 56.24% | 0.68108 |
| Transformer, no rank | 58.90% | 0.66831 |
| Transformer, rank-conditioned | 59.14% | 0.66703 |
Conditioning on rank is worth +0.247 accuracy points, and the rank-conditioned run is ahead at every single one of the eight epochs rather than sneaking ahead at the end.
The bit I enjoyed more than the result: I knew roughly what it would be beforehand. Take that per-bracket table, compute each bracket's gain from being fitted within itself instead of pooled, weight by how much of the population that bracket holds, and you get +0.256 points. The Transformer delivered +0.247.
Those are genuinely different mechanisms -- seven separate linear fits on one side, one network reading a rank embedding on the other -- so this is a consistency check rather than a real prediction. But it's the first time in this project that a cheap descriptive measurement has told me in advance what an expensive model was going to do, and after four months of being surprised by my own training runs, I would like to record that it felt fantastic.
The Sentence in the Logs I Didn't Want to Read
Both of those runs picked their final epoch as their best. Validation log loss at epoch seven was 0.66731; at epoch eight it was 0.66703. Still falling.
Which means 59.14% isn't what this architecture achieves. It's what this architecture had reached when I stopped asking. My cosine schedule was set to eight epochs because eight epochs was about forty minutes and forty minutes was about how long I was willing to wait.
And that's the actual state of this project, stated plainly: I have been choosing hyperparameters by vibes since March. Learning rate, embedding width, layer count, weight decay, schedule length -- every one of them is a number I typed once, and every one of them I have subsequently defended on the grounds that the model trained fine.
The fix is a hyperparameter search. Not a clever one. Any one.
So I worked out what that costs.
Each run was a bit under forty minutes for eight epochs, and I now know eight is too few, so call it 24: 1.95 hours a trial. Thirty trials is a small search by any standard, and thirty trials is 59 hours. Two and a half days of a GPU doing nothing else, to answer questions I should have been asking since spring.
That's the post. Not the search -- the search is Part 7. This is the week I spent making the search cost something I was willing to pay.
There turned out to be exactly three levers, and none of them are about the model.
Stop Computing Things So Precisely
Neural networks are, by default, trained in 32-bit floating point. Every weight, every activation, every gradient: 32 bits.
The observation behind mixed precision is that most of that is waste. Sixteen-bit floats have about three decimal digits of precision, which is plenty for an activation, and modern GPUs have dedicated silicon -- tensor cores -- that multiply 16-bit matrices roughly four times faster than 32-bit ones. My GPU has had them the entire time. I have never used them.
The reason you can't just cast everything to fp16 is range, not precision. The smallest positive number fp16 represents normally is about 6e-5. Gradients are frequently smaller than that. Cast naively and a meaningful slice of your gradients round to exactly zero, which is not a slightly worse update -- it's no update, on whatever those parameters were about to learn.
The fix is charming in its bluntness. Before running backpropagation, multiply the loss by a big constant: 65,536, say. Every gradient is scaled up by that same factor, straight into the range fp16 represents comfortably. Then divide them all back down before the optimizer touches them. This is what PyTorch's GradScaler does, and it picks the constant adaptively: push it up while things are fine, and if a gradient ever overflows to infinity, throw that step away and halve it.
Weights stay in fp32 the whole time. Only the forward and backward passes go fast and loose.
There is one place this bites, and it bit me. I clip gradients by norm. If you clip while the gradients are still scaled, you clip against a threshold 65,536 times too small, which quietly turns "clip the occasional outlier" into "crush every update." The scaler has to be told to unscale first:
scaler.scale(loss).backward()
scaler.unscale_(optimizer) # <- without this, the clip below is nonsense
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm)
scaler.step(optimizer)
scaler.update()
Two lines in the right order. The result, timed per epoch across two runs identical in every hyperparameter, on a GPU I was careful not to touch while they ran:
293.1 seconds per epoch became 138.6. A 2.12x speedup.
I checked it hadn't broken anything, because a fast wrong answer is worse than a slow right one. Epoch one, full precision: 58.45% accuracy, 0.670873 log loss. Epoch one, mixed precision: 58.47%, 0.670687. Fine.
Stop Babysitting Runs
The GPU lives in a headless server. That's the right place for it, but it meant every experiment was an attended operation: open a session, set up the environment, launch the run, keep the connection alive, come back later to read the log.
Written down that sounds like nothing. In practice it meant I only started a run when I had a block of time to sit with it, which meant I ran experiments in batches, which meant I ran fewer of them, which meant each one had to be a good idea in advance -- which is precisely the mindset a hyperparameter search exists to replace.
I already had a distributed job scheduler from earlier in this series. It had been sitting idle since Part 3. So I pointed it at the training script, gave the worker a token, and now experiments start from wherever I happen to be working:
python -m research.schedule submit --section Tower --T_max 24 --amp 1
The scheduler derives each worker's identity from a hash of the hyperparameters it advertises, which is a nice property -- change the search space and you get a visibly new worker rather than a silently mismatched old one.
It is also how I lost most of an evening.
I'd added the --amp flag, pulled on the GPU box, confirmed the pull, and checked that the worker had picked up the new option. It hadn't. Same thirteen parameters, no amp, heartbeat perfectly healthy and current. I pulled again. I checked the commit hash. I checked I was on the right branch. I stared at a perfectly correct git log for quite a while.
The worker reads the training script's parameter list once, at startup, and then reports that cached copy in every heartbeat forever. The process was still the one I'd launched before the pull. The code on disk was new; the code in memory was four hours old. Restarting it fixed everything instantly.
The lesson isn't "restart your services." The lesson is that I spent an hour debugging the deployment and zero seconds asking what the heartbeat was actually made of. A green status light is a claim, and it's worth occasionally knowing which claim.
Stop Searching Stupidly
Thirty trials of what, though.
Grid search is out immediately. Ten hyperparameters at three values each is 59,049 runs, and it spends most of its budget re-measuring parameters that don't matter. Random search is genuinely decent and is what I'd have reached for, but it has no memory: trial thirty knows nothing that trials one through twenty-nine learned.
The thing I'd been vaguely aware of and had never actually understood is Bayesian optimization, and specifically the flavor Optuna uses by default: the Tree-structured Parzen Estimator.
The intuition took me embarrassingly long to get, mostly because I assumed it was doing something harder than it is. Here's the whole idea.
Run some trials. Sort them by result. Draw a line -- the top quantile is "good," everything else is "bad." Now fit a probability distribution over hyperparameter values to each group separately: call l(x) the distribution of values that produced good results, and g(x) the distribution of values that produced everything else.
Then propose whichever x maximizes l(x) / g(x).
That's it. Not "where do I predict the score is highest" -- TPE never models the score at all. It models where the good trials live and the bad ones don't, which is a much easier question, and it answers it with a ratio of two densities you can fit in microseconds.
A Gaussian process -- the other standard approach -- builds a full surrogate of the objective surface and asks where that surface is high and uncertain. It's more principled, and it scales badly with the number of trials and awkwardly with categorical parameters like "how many layers." TPE handles a mixed continuous-and-categorical space naturally and costs almost nothing to run, which is the correct set of trade-offs for someone whose real bottleneck is a single consumer GPU.
The "tree-structured" part is about conditional parameters -- decoder width only means something if you have a decoder -- which I don't need yet but will the moment I start varying architecture.
What It All Bought
Three levers: each run is 2.1x cheaper, starting a run costs a command instead of an evening, and the runs are chosen by something with a memory.
Only the first is a number, so here's that number honestly:
Twelve trials a day became twenty-five. The thirty-trial search that was going to cost me two and a half days now costs a little over one.
That's not a dramatic result. It is, however, the difference between a search I keep deciding to run next weekend and a search I can start tonight and read over breakfast -- and those are not the same object, no matter how similar they look on a spreadsheet.
Takeaway
The engineering lesson is one I apparently need to relearn every few months in a new costume. In Part 5 it was optimizing a data collector for a month without measuring the ceiling of the approach. This time it's subtler and more embarrassing: I let the cost of an experiment silently set the shape of my research.
Eight epochs wasn't a considered choice. It was forty minutes, which was my patience, which then became a hyperparameter, which then became the number that every model in this series has been quietly trained against. Nobody decided that. It accreted.
The tell was sitting in my logs the whole time. Best epoch: 8, of 8. Every run. For months. A model that always improves right up to the moment you stop it isn't telling you it's finished -- it's telling you that you are the binding constraint.
Next time: what I'm actually going to search over. Because it turns out that while I was making training cheap, the more interesting problem was waiting underneath it -- my model has only ever seen finished drafts, and a draft recommender is by definition a thing you use on an unfinished one. That gap is the whole product, and closing it needs a different objective, a different architecture, and a control arm I nearly forgot to include.
That one has real results in it. This one just made them affordable.
One loose thread from that bracket table that I have no explanation for.
The most predictable bracket in Dota is not the bottom one. It's Guardian, the second -- 57.17%, a clear third of a point above Herald's 56.82%. The curve goes up before it comes down.
Every story I can tell about why drafts predict outcomes says Herald should be the peak. Least mechanical skill to override a bad matchup, most games decided by composition. Instead there's a bump.
The explanations I can think of are all about Herald being a strange place rather than a simple one -- brand-new accounts, smurfs on their way up, people who queue once a year -- and I can't currently distinguish between any of them with data I have. So it's just sitting in my notes, being odd.
I've learned to like these. The last one of these I wrote down was "Bristleback seems rank-dependent," and it turned into the entire first half of this post.