My rate limiter had failed silently three times before I tested the right thing

A rate limiter on one of my sites has failed open before. Silently, three separate times, spread far enough apart that I never connected the pattern until this week.

Here is the failure mode. The limiter depends on a Cloudflare KV binding, referenced by name. If that name is ever wrong, misconfigured, or dropped during a config change, the check that is supposed to block abusive traffic simply does not run. There is no error. There is no log line. Every request just succeeds, exactly as if the limiter did not exist.

I already had two tests guarding this code. A unit test for the limiter’s logic, which always passed, because it always handed the function a working fake binding to test against. And a synthetic monitor that hit the live signup endpoint once, which also always passed, because a single request can never trip a rate limit regardless of whether the limiter is doing its job.

Both tests were checking a true thing. Neither test could have caught the specific failure that had already happened three times, because neither one tested whether the real binding, in production, was actually there and actually working.

That is the gap: a test that proves your logic is correct is a different claim from a test that proves your logic is running. I had built plenty of the first kind and none of the second.

So this week I wrote a new check, and it is deliberately simple. It sends the real configured limit’s worth of requests to the live endpoint, sequentially, then sends one more. If any request inside the limit fails, something upstream is broken. If the request past the limit is not rejected, the limiter is failing open right now, in production, with nothing else positioned to notice.

I ran it for real against the live site. Ten requests through clean. The eleventh came back rejected with a 429. I wired the check into a daily job that pages me the moment it ever comes back wrong.

The general version of this lesson applies past rate limiters. Any safety check that depends on an external binding, a feature flag, an environment variable, a config value pulled from somewhere else, needs a test that exercises the real dependency, not a stand-in for it. A mock proves your code behaves correctly when the dependency is present and working. It says nothing about the day the dependency is quietly gone.