Every week, I host a Magic: The Gathering draft night for a few friends. As the person who got everyone into the hobby, I feel a certain… custodial responsibility. The draft experience must be perfect. This has led to a borderline obsessive post-draft ritual: spending hours on Scryfall, hunting for new cards, and agonizing over which underperforming cards in my cube should get the axe.

The process was slow, deeply subjective, and prone to my own biases. As any self-respecting engineer would do when faced with a manual, inefficient process, I decided the answer was to throw an unreasonable amount of code at the problem.

What started as a simple utility script descended into a delightful rabbit hole of performance tuning, architectural debate, and ultimately, a fully-fledged web application that I’m genuinely proud of. The project is now live at mtg.scottliu.com/cube/recommend.

This is the story of that engineering journey.

Phase 1: The Quest for Objective Truth (and a Working Scraper)

The initial goal was simple: create a "composite score" for every Magic card based on three pillars:

  1. Pick Rate (The Hype Factor): How often is this card actually picked by the global cube community?
  2. Word Count (The Simplicity Factor): Can my friends read this card without needing a PhD in rules lawyering?
  3. Price (The "Can I Remortgage My House?" Factor): Can I actually afford this piece of cardboard?

My first instinct for getting pick-rate data was a simple web scraper. However, a quick look at my browser's network tab revealed a much cleaner solution: a hidden JSON API the website was using internally. This was the first critical insight of the project. By targeting the API directly, the data pipeline became far more robust. It's a classic engineering lesson: always check for an API before you start parsing HTML.

Of course, no new project is complete without a ceremonial bug. The Scryfall bulk data downloader immediately crashed with a ZeroDivisionError -- a classic so common it might as well be scripted. The server, in its infinite wisdom, had decided not to tell me the file size. You know what they say: "infinite gains!" A quick patch later, I had a working, if clunky, Python script that could spit out a sorted list of cards to my terminal.

Phase 2: The "Black Lotus" Incident

A wall of text in a terminal is functional, but the first set of recommendations revealed a hilarious flaw. The script, with cold, robotic logic, proudly suggested I add the Power 9 to my cube. Its reasoning? They had a perfect affordability score.

Why? Because for cards like Black Lotus, which are traded in back alleys for small fortunes, the price field in the data is simply NULL. My script, bless its heart, interpreted this as "free." This was a keen reminder that in data-driven systems, the absence of data is itself data. I had to teach my creation that some cardboard is so expensive, it transcends mere numbers. A quick fix to filter out NULL prices solved the problem and made the recommendations instantly more realistic.

It was time to build a real application. I spun up a Flask server and made the foundational decision to use SQLite as a data store. Parsing a 155MB JSON file of every Magic card on every request was a non-starter; a one-time bootstrap process was the only sane path forward.

Phase 3: The Great Architectural Debate

The interactive web UI was working, but there was a problem. The "Adds" page, which had to filter over 30,000 cards against my long list of banned keywords, took over a second to load. For an engineer, a 1-second load time isn't a feature; it's a bug that needs to be hunted.

This led to the great architectural debate, a classic engineering dilemma I wrestled with for the better part of an afternoon.

Approach #1: The Just-in-Time Engine This was my first major insight into a truly robust solution. Instead of a clumsy heuristic, I could build a sophisticated SQL subquery. The query would tell the database: "First, perform all the fast filters (price, color, mana cost) and sort the results to get a candidate pool of the top 500. Then, apply the slow text filters to that much smaller, pre-qualified set."

This was already pretty good. It was fast, perfectly accurate within the generous 500-card buffer, and a massive improvement. The "Adds" query time dropped from 1000ms to a snappy 280ms. But I knew I could do better. The system felt... incomplete.

Approach #2: The Pre-Computed Universe The alternative was to pre-calculate the final score for every single card during the bootstrap phase. This would make the final queries instantaneous (ORDER BY add_score DESC). The downside? The bootstrap process would take ages, and every time I wanted to tweak my scoring formula, I'd have to re-run the entire thing. The feedback loop for tuning would be painfully slow.

I was torn between two seemingly exclusive solutions: the fast-but-rigid pre-computed model, and the flexible-but-slower just-in-time model.

Phase 4: The "Aha!" Moment - The Hybrid Engine

And then, the lightbulb moment. Why not have both?

This was the final architectural breakthrough that tied the whole project together. I could build a hybrid engine that intelligently switched between the two modes.

  1. Formula Hashing: The score_calculator.py module now generates a unique "hash" (a digital fingerprint) of all its scoring knobs.
  2. Pre-computation: The bootstrap script pre-calculates all scores and stores this formula hash in the database.
  3. The Magic: When the web app runs, it compares the hash of the current formula in the code to the hash stored in the database.
    • If they match, it uses the "fast path," pulling the instantaneous, pre-computed scores.
    • If they don't match, it prints a warning and automatically falls back to the perfectly accurate, real-time "Live Tuning Mode."

It required a full rewrite of the database schema and the analysis logic, but it was entirely worth it. The system is now the best of all worlds. The live, deployed version is incredibly fast, while my local development version provides immediate feedback whenever I tweak a formula, without ever needing to re-bootstrap.

The Final Product

What started as a weekend hack to improve my draft nights has become a fully-fledged, performant, and intelligent web application. It’s a testament to the power of iterative development, the importance of questioning your own assumptions, and the sheer joy of over-engineering something you're passionate about.

And if you're in my playgroup... be ready. The optimization never stops.