When we last left our hero, the model had finally learned something. It's not much, but it's honest work: 56.27% test accuracy, after I figured out my entire data-quality hypothesis was backwards and started training exclusively on Herald-through-Archon matches. That number is a hard-won three points above the dumb baseline. It was, after weeks of debugging, the first time I felt I had a model that was actually doing the task I asked it to do.
So obviously I wanted to make it learn something more. The plan, as with all machine learning plans at this stage, was to throw hyperparameters at it. Tweak the learning rate. Tweak the weight decay. Try a smaller embedding dimension. Try a larger one. Light a candle, sacrifice a goat. The usual.
I started by submitting one training job from the command line. That's where this post really begins.
Part 1: The Coffee Run
Here's the thing about a 25-minute training run. The first time you submit one, you sit at your desk and watch every line of train_loss scroll by, because you have no idea what could go wrong and you absolutely do not want to be away from the keyboard if something does. So the first job, I just watched.
The second job, I went and got coffee.
The third job, I went and got coffee, came back, did some emails, realized the job had finished, edited the config to bump the learning rate up by 0.0005, kicked off another, and went to get another coffee.
That was the moment I noticed I was about to spend a Saturday opening Terminal, editing a config, hitting up arrow, hitting enter, walking away, coming back, copying a number into a notepad, repeat. And not just one Saturday -- probably several.
I have a rule, by the way. The rule of three. The first two times I do something repetitive, I let it slide -- everyone gets a freebie, and if it really is just twice it might be cheaper to do it twice than to build the abstraction. The third time, though? That is when DRY shows up at the door, glaring at me, and I have to admit it's right.
This was the third time. And I had not really started yet.
Part 2: The "But There's a Tool For That" Aside
Yes. I know. I am aware.
I'll spare us all the brand names, but there is an entire shelf of extremely good, extremely battle-tested industry tools for tracking ML experiments and queueing hyperparameter sweeps. A professional version of me, on a deadline at work, would absolutely reach for one of them without thinking twice. That's the whole point of industry standards -- somebody else has already eaten the bug fixes for you, and on a deadline that matters more than understanding every layer of the stack.
But this is not work. This is the hobby. The whole point of a hobby project is that you get to pick which battles you fight. When something goes wrong in a system I built, I know exactly where to look. When something goes terribly wrong, I can pivot the entire architecture in an afternoon, because nobody is paying for the previous architecture except me. That latitude does not really exist in a professional setting, and I think it's one of the genuinely best parts of being an engineer by hobby in addition to by profession.
So. I was going to build my own little scheduler. Of course I was.
Part 3: The Self-Describing Machine
Before writing a single line of code, I sat down and wrote a design doc. I know how that sounds -- but for hobby projects, especially ones that involve two services talking to each other across a network, spending a couple hours getting the API contracts and state machine right up front saves an order of magnitude more time later. The first commit in the scheduler repo is literally just a 500-line markdown file describing the system. The actual code did not show up until the next day.
The system needed two pieces:
- A scheduler. A Flask app with a web UI. I submit jobs through a form, watch them run on a dashboard, pause or cancel them mid-flight if I change my mind. State persists in SQLite.
- A worker. A small Python process running on my GPU machine. It polls the scheduler for new jobs and runs them.
Most of this is unsurprising. The plumbing is REST APIs, JSON payloads, a state machine for jobs (PENDING -> IN_PROGRESS -> COMPLETED, with side branches for PAUSED, FAILED, and CANCELLED). All standard.
There is one design decision I'm actually proud of, though, and it's the one that took the longest to get right in my head before any code was written.
The worker tells the scheduler what hyperparameters exist.
That sentence sounds boring. Let me show why it isn't.
In a naive design, the scheduler's web UI hard-codes the list of hyperparameters: learning rate, batch size, dropout, number of layers. Every time I want to expose a new hyperparameter from the training script, I have to update the training script and update the scheduler UI, and then deploy both. That's two places where the same piece of information lives, and you know how that goes -- they will drift, one of them will be wrong, and I will spend a frustrated evening figuring out why.
The fix: the training script declares its own schema. When you call it with --print_schema, it prints out something like this:
{
"max_lr": {
"type": "float",
"default": 0.001,
"min": 0.0001,
"max": 0.01,
"name": "Maximum Learning Rate"
},
"n_layers": {
"type": "int",
"default": 4,
"choices": [2, 3, 4, 5, 6],
"name": "Transformer Layers"
}
}
The worker reads that on startup, sends it along with every poll request, and the scheduler stores it. The web UI then renders its form entirely from that schema -- input ranges, dropdowns, default values, even the human-readable labels. If I add a new hyperparameter to the training script, the next time the worker polls, the dashboard has a new input field. No deploy. No second source of truth.
The scheduler does not actually know or care that this is a Dota draft model. It could be classifying handwritten digits tomorrow, or scoring Magic: The Gathering cubes, or anything else I want to point a worker at. That generality was deliberate -- I wasn't building this for one project, I was building it for every future ML project I might ever poke at. Locking the UI to a hard-coded schema would have undone all the leverage.
Part 4: A Day of Building, A Day of Bugs
The first commit was the design doc. The next day, I sat down and built the whole thing in an afternoon. The git log from that day is admirably honest about what that looked like:
4130a9e Added better logging, removed some jsonify
c48ef8f Replaced the ON CONFLICT which was causing issues
d76b7e0 Fixed error when submitting jobs
e21b243 Store logs on failure and display
You can read the entire arc in those commit messages. Submit a job. It crashes. Add more logging. Discover the SQLite ON CONFLICT clause is silently swallowing duplicate sections instead of updating them. Replace it. Submit again. The job runs but the worker errors are vanishing into the void on failure. Capture the last fifty lines of stderr and stash them in the database so I can see them on the dashboard. Try again. It works.
The pause/cancel plumbing turned out to matter more than I expected. The worker checks the job status every five seconds during training, so if I hit Cancel from the browser, the worker notices, terminates the training subprocess, and reports back. Same for pause and resume. Once I started queueing eight or ten jobs at a time, sometimes I'd realize halfway through one that I'd given it a stupid configuration -- and being able to kill it from the dashboard without SSHing in was a small but real luxury.
I also kept tweaking the UI for days after the backend stopped changing. Inputs that don't reset when the page polls. A subtle highlight on any value that differs from the default. Hiding sections from workers that have not checked in lately. Each of those was the kind of paper cut you only notice once you actually start using the thing in anger. The dashboard kept teaching me what it was missing.
Part 5: Tower and the Tunnel
Quick infrastructure note. The scheduler lives on my cloud VPS, where the rest of my personal sites already run. The training has to happen on a desktop machine I have at home -- the one with the actual GPU. I'll call it Tower, because a long time ago I had an internship where the company's remote GPU desktops were all named tower1, tower2, tower3, and apparently that lodged itself in my head as the default name for "a desktop that does the heavy lifting."
The problem is that Tower lives behind my home NAT, and the scheduler lives in the cloud, and you can guess which way the network connection has to go.
If you've read my previous posts, you already know how this story ends. autossh. A persistent reverse SSH tunnel from Tower out to the cloud VPS, so the worker can reach the scheduler over a stable connection without me having to go anywhere near my home router. I'm not going to relitigate the whole pattern here -- I wrote a whole post about it -- but there was something quietly satisfying about reaching for infrastructure I'd already built for an unrelated reason and finding it was exactly the right shape for this. Reuse, in the noblest sense.
Part 6: The Morning After
I queued five jobs the first night. Different learning rates, different weight decays. I went to bed.
The next morning, I opened the dashboard and looked at five rows of completed jobs, each with a test accuracy and a training time and a path to a saved model checkpoint. Tower had churned through them one after another while I slept.
I want to say what I felt was efficiency. It would be on-brand for an engineer to feel "ah yes, time saved, optimization complete, productivity unlocked." But that's not really what it was. What it was was a quiet, surprisingly genuine satisfaction at having designed something coherent on paper, having pieced it together from a bunch of moving parts, and then having it actually work the way the design doc said it would.
Efficiency, as it turns out, was the boring part. The interesting part was the journey -- and that's a lesson that keeps showing up the older I get. The point of the hobby was never to find the optimal Dota 2 draft. The point was to spend a weekend designing something I fully understood, end to end, exactly the way I wanted it.
Also, now I had a job queue.
What's Next
The rest of the dashboard, in case you're wondering, is what happened over the four days that followed. I queued up batches of experiments, walked away, queued up more, and slowly watched a column of test accuracy numbers crawl up from 56% toward something more respectable. By the time I called it, the scheduler had logged 73 completed jobs, several minor breakthroughs, one architectural pivot I genuinely did not see coming, and a final winning configuration that surprised me.
That's Part 4.
For now, my model still has no idea which hero counters which. But it's failing to figure that out in parallel -- on a schedule, with the results plotted neatly in a browser tab while I sleep. We'll call that progress.