I'd recently written my own authentication service and my own metrics service, rather than using anything off the shelf. Every login and every data point went through code I'd written and understood completely.

Which is how I managed to build two services that could not start without each other.

The Paradox: The Guard Who Locked His Keys in the Office

The problem revealed itself under the most mundane of circumstances: I wanted my shiny new Authentication Service to emit metrics. How many login attempts per minute? How long does token verification take? Simple questions. My Metrics Service was ready and waiting.

Here's how the flow was supposed to work:

  1. A user attempts to log in, calling the Authentication Service.
  2. Before responding, the Authentication Service makes a quick call to the Metrics Service to log the event ("one login attempt!").
  3. The Metrics Service records the data. Simple.

But here's where the paradox slammed the door shut. To prevent just anyone from flooding it with junk data, my Metrics Service was designed to be secure. And how does it secure itself? By calling the Authentication Service to verify the incoming request has a valid token.

You see the loop:

  1. Auth Service gets a login request and tries to call the Metrics Service.
  2. Metrics Service receives the call and, to validate it, turns around and calls the Auth Service.
  3. Auth Service receives this new validation call and... tries to emit a metric about it, calling the Metrics Service again.
  4. Goto 2.

An infinite loop, with both services waiting politely on each other forever. I had built a security guard who needs to swipe his ID to get into the building, and whose ID card is on his desk, inside the building.

The Investigation: Searching for an Escape Route

To break the deadlock, I had to find a way to let the guard into the building. I reasoned through two main contenders.

Contender #1: The Trusted Fortress

The first idea was to create a security exception based on network location. I could configure the Metrics Service to say, "If a request comes from another machine within our private, trusted network, just let it through. No questions asked."

It would work. It would break the loop. But it felt wrong. It would tightly couple my services to the underlying network hardware. If I ever wanted to move the Metrics Service to a different cloud or hosting environment, this rule would break. It violated a core value: build modular systems, not hardware dependencies. It felt like tying two ships together just because they were in the same harbor. I discarded it.

Contender #2: The Skeleton Key (The Internal Token)

The next approach was to create a special, hard-coded "skeleton key": a secret token the Auth Service would use when calling the Metrics Service. The Metrics Service would have a small piece of logic at the top: "If you see this specific token, don't call the Auth Service. Just let it in."

This would work. It breaks the loop cleanly. But it felt so... inelegant. It meant creating and managing a separate class of secrets, special back-door keys that bypass the very system I was trying to build. Every time I spun up a new service, would I need a new special token? The thought of managing a growing collection of these one-off keys gave me a headache. It was a fix, but it wasn't a pattern I wanted to replicate.

The "Aha!" Moment: It's a Problem of When, Not Who

I implemented the skeleton key for the time being, but the problem gnawed at me. I kept staring at the diagram of the loop, trying to figure out how to break the circle.

Then it clicked.

The circular dependency wasn't in the logic. It was in the timing.

The Auth Service wasn't just calling the Metrics Service; it was waiting for it to respond before it could finish its own work. The synchronous nature of the HTTP call was the real villain.

What if it didn't have to wait?

Change the guard's job description. He doesn't need to get confirmation before he enters the building. He just needs to sign a logbook on his way in, confident that an auditor will review it later. So Auth buffers its metrics in memory, and a background thread ships them after the response has already gone out. No request path ever blocks on the Metrics Service again.

That was the reframe, and it was the right one.

It is also, by itself, not sufficient -- which took me considerably longer to admit than it took to notice.

The Part Asynchrony Doesn't Fix

Asynchrony breaks the deadlock. It does not break the cycle.

Walk the loop again with buffering in place. Auth handles a login, buffers a metric, responds immediately. Good. Ten seconds later the flush thread wakes up and POSTs that metric to the Metrics Service. Metrics receives the call and, to authorize it, calls Auth. Auth serves that authorization request and, being an instrumented service, buffers a metric about having served it. Ten seconds later, the flush thread wakes up.

Nothing deadlocks. Nothing blocks. Nothing stops, either. I would have traded an infinite loop for an infinite loop with better manners -- a steady state where every published metric generates roughly one more metric to publish, and two services discuss, forever, the fact that they are talking to each other.

So there has to be a real boundary somewhere. There is one. It just isn't the one I was giving myself credit for.

Auth doesn't emit metrics through the shared client that every other service uses. It has its own small module, and that module posts to a different endpoint:

POST /api/internal/publish-auth-metrics

That route does three things, and the first one is load-bearing.

It performs no authorization check, so the Metrics-to-Auth leg is never traversed for Auth's own metrics. It rejects any payload whose metric name doesn't start with AuthService.. And it is uninstrumented -- alone among the routes in that service, nothing decorates it, so recording an auth metric does not generate a metric about recording an auth metric.

Those three properties are what terminates the recursion, and the first is doing most of the work. Every other service in the system still publishes through the authenticated endpoint exactly as designed. The cycle is cut at precisely one edge: the one that closes it.

Which means the honest accounting is that I did not replace the skeleton key. I kept it. Asynchrony solved a different problem that I had spent two evenings conflating with this one.

In fairness to past me, it is a much better skeleton key than Contender #2. That one was a secret: whoever held it could write anything, and every new service would eventually want one. What shipped is narrower on every axis. There's no credential to leak, because there's no credential. There is exactly one route with this property, and it doesn't generalize -- a second service wanting the same treatment would need its own route, its own namespace prefix, and a fresh argument for why it deserves one. Attaching a namespace to the exception is the difference between a back door and a mail slot.

It still costs something, and I'd rather write the cost down than rediscover it in a code comment. That route accepts writes with no credential, and the host it lives on -- metrics.internal.scottliu.com -- is a public DNS name that resolves for anyone who asks. "Internal" there means bypasses my CDN, not unreachable from the internet. So anybody who knows the path can write nonsense into the AuthService.* namespace. They cannot read anything, cannot touch any other metric name, and cannot obtain a credential from it. The blast radius is "my login-count graph is wrong," which on a personal service is a trade I'll take -- but it is a trade, and the thing that bounds it is the namespace check, not the word "internal."

What It Bought

The namespace-scoped route is what makes the system correct. Separating recording a metric from delivering it is what makes it good, and two things came with it:

  • Performance: API latency became minimal and predictable, completely insulated from the performance of the Metrics Service.
  • Resilience: An outage in the Metrics Service now has zero immediate impact on the Auth Service, which continues to buffer metrics until the downstream service recovers.

I'd spent two evenings hunting for the right exception to my own security rule, decided the rule was fine and the waiting was the problem, and shipped the exception anyway without noticing I'd shipped it. Both things were true. Only one of them was the answer to the question I'd asked.

The tell was that I never went back and deleted the skeleton key. I told myself the timing fix had made it unnecessary, and then left it exactly where it was. My code had a better model of the problem than I did.

Worth noting what I gave up, though: metrics are now eventually consistent. If the Auth Service dies with a full buffer, those events are gone. For counting logins that's an easy trade. For anything I needed to bill on, it wouldn't be.