Every week, I host a Magic: The Gathering draft night for a few friends. As the person who got everyone into the hobby, I feel a certain... custodial responsibility. The draft experience must be perfect. This has led to a borderline obsessive post-draft ritual: spending hours on Scryfall, hunting for new cards, and agonizing over which underperforming cards in my cube should get the axe.

The process was slow, deeply subjective, and prone to my own biases. As any self-respecting engineer would do when faced with a manual, inefficient process, I decided the answer was to throw an unreasonable amount of code at the problem.

What started as a simple utility script turned into weeks of performance tuning, one long argument with myself about architecture, and eventually a web application I'm genuinely proud of. The project is now live at mtg.scottliu.com/cube/recommend.

This is the story of that engineering journey.

Phase 1: The Quest for Objective Truth (and a Working Scraper)

The initial goal was simple: create a "composite score" for every Magic card based on three pillars:

  1. Pick Rate (The Hype Factor): How often is this card actually picked by the global cube community?
  2. Word Count (The Simplicity Factor): Can my friends read this card without needing a PhD in rules lawyering?
  3. Price (The "Can I Remortgage My House?" Factor): Can I actually afford it?

My first instinct for getting pick-rate data was a simple web scraper. However, a quick look at my browser's network tab revealed a much cleaner solution: a hidden JSON API the website was using internally. Going straight at the API meant I was reading structured data instead of guessing at HTML that could be redesigned out from under me at any time. Always check for an API before you start parsing HTML.

Of course, no new project is complete without a ceremonial bug. The Scryfall bulk data downloader immediately crashed with a ZeroDivisionError: the server had declined to tell me the file size, and my progress bar divided by it anyway. A quick patch later, I had a working, if clunky, Python script that could spit out a sorted list of cards to my terminal.

Phase 2: The "Black Lotus" Incident

A wall of text in a terminal is functional, but the first set of recommendations revealed a hilarious flaw. The script, with cold, robotic logic, proudly suggested I add the Power 9 to my cube. Its reasoning? They had a perfect affordability score.

Why? Because for cards like Black Lotus, which are traded in back alleys for small fortunes, the price field in the data is simply NULL. My script, bless its heart, interpreted this as "free." A quick fix to filter out NULL prices solved the problem and made the recommendations instantly more realistic.

It was time to build a real application. I spun up a Flask server and made the foundational decision to use SQLite as a data store. Parsing a 155MB JSON file of every Magic card on every request was a non-starter; a one-time bootstrap process was the only sane path forward.

Phase 3: The Great Architectural Debate

The interactive web UI was working, but there was a problem. The "Adds" page, which had to filter over 30,000 cards against my long list of banned keywords, took over a second to load. For an engineer, a 1-second load time isn't a feature; it's a bug that needs to be hunted.

I wrestled with it for the better part of an afternoon.

Approach #1: The Just-in-Time Engine Instead of a clumsy heuristic, I could push the work into a SQL subquery. The query would tell the database: "First, perform all the fast filters (price, color, mana cost) and sort the results to get a candidate pool of the top 500. Then, apply the slow text filters to that much smaller, pre-qualified set."

This was already pretty good: fast, and accurate within the 500-card buffer. Query time dropped from 1000ms to 280ms. But I knew I could do better.

Approach #2: The Pre-Computed Universe The alternative was to pre-calculate the final score for every single card during the bootstrap phase. This would make the final queries instantaneous (ORDER BY add_score DESC). The downside? The bootstrap process would take ages, and every time I wanted to tweak my scoring formula, I'd have to re-run the entire thing. The feedback loop for tuning would be painfully slow.

I was torn between two seemingly exclusive solutions: the fast-but-rigid pre-computed model, and the flexible-but-slower just-in-time model.

Phase 4: The "Aha!" Moment - The Hybrid Engine

Then it occurred to me: why not have both?

I could build an engine that switches between the two modes on its own.

  1. Formula Hashing: The score_calculator.py module now generates a hash of all its scoring knobs.
  2. Pre-computation: The bootstrap script pre-calculates all scores and stores this formula hash in the database.
  3. The Check: When the web app runs, it compares the hash of the current formula in the code to the hash stored in the database.
    • If they match, it uses the "fast path," pulling the instantaneous, pre-computed scores.
    • If they don't match, it prints a warning and automatically falls back to the perfectly accurate, real-time "Live Tuning Mode."

It required a full rewrite of the database schema and the analysis logic. Worth it. The deployed version is fast, and my local copy gives me immediate feedback whenever I tweak a formula, without re-bootstrapping anything.

The Final Product

A weekend hack to improve my draft nights turned into a real web application, which is roughly the ratio of effort to necessity I aim for in a hobby.

The part I'd actually take to work with me is the Black Lotus bug. A missing price wasn't missing data -- it was the most important data in the row, and my code read it as zero. I go looking for that shape of mistake everywhere now.

And if you're in my playgroup... be ready. The optimization never stops.