Our Site Said "<25ms Overhead." We Could Not Prove It. Measuring Found Three Bugs.

Gateco is a policy enforcement layer that sits between AI agents and vector databases: every retrieval is evaluated against access policies before any content is returned. For months, our website said this added "under 25ms p95 policy-layer overhead," and in one place it said the figure came from production measurements.

Last week, during a competitive review, someone asked a simple question: measured where, by whom, against what baseline? We went looking for the artifact. There wasn't one. No benchmark script, no recorded run, no CI gate, and, as it turned out, no way to even compute the number from our telemetry. The figure was plausible engineering intuition that had hardened into a published claim. So we measured properly. This post is what happened next.

The first honest number was 302ms

We wrote a benchmark to match the methodology our own docs page described: a pgvector connector with 1,000 embeddings (1536-dim, HNSW), 100 registered resources with classification and sensitivity metadata, one active five-rule RBAC policy, one synced principal, top_k=10, client-supplied query vectors so server-side embedding stays out of the measurement. A hundred and fifty sequential requests per run against a quiet dev backend, after warmup.

Policy-layer overhead, p95: **302 milliseconds**. Twelve times the published claim.

The interesting part is why. The gap wasn't measurement noise or a slow policy engine. It was three real bugs, and none of them were where we expected.

Bug one: the documented policy selector matched nothing

Every benchmark request came back with zero allowed chunks and forty denied. Our policy allowed roughly 70 percent of the corpus, so something was structurally wrong.

The benchmark scoped its policy to one connector using the selector form documented in our own API schema: `{"field": "connector_id", "op": "eq", "value": "<uuid>"}`. It turned out that form had never worked. The resource model stores the connector reference under a different attribute name, and the generic field lookup returned nothing; even when it could resolve a value, it compared a UUID object against a JSON string. The selector evaluated false for every resource, so the policy applied to nothing, and our deny-by-default engine denied everything the policy was supposed to allow.

Two things are worth saying about this. First, it failed closed: the bug produced over-denial, never exposure. Under-granting is the failure mode you want in an authorization system, and this is exactly why. Second, we checked the blast radius before writing this post: production had zero policies deployed, so no customer ever hit it. The bug lived only long enough for our own benchmark to be its first victim, which is precisely what the benchmark was for.

A shorthand selector form worked fine, which is why our integration suites never caught it. The fix is a few lines plus three regression tests that pin the documented form to the documented behavior.

Bug two: the overhead was unmeasurable by construction

Our plan for backing the claim long-term was to compute overhead from per-retrieval telemetry: every retrieval record stores its total service time and its connector (vector query) time, and the difference is the policy layer. Except for vector search, the most common mode by far, the connector time was never recorded. The search dispatcher stamped the latency for keyword and hybrid modes and returned early for vector mode. Every vector retrieval in the audit trail had a connector time of zero.

So the honest statement is not just "the claim was unverified." It's "the claim was unverifiable from the data we kept." One line of code and one test later, every search mode records its split, and the overhead figure is now computable from any deployment's own audit records.

Bug three: the pipeline paid for a second search on almost every request

With the selector fixed, overhead dropped to 256ms p95. Still terrible. Per-step timing showed two blocks of ~125ms each: the connector query itself, and a block labeled "evaluation and refill."

The refill loop is a feature: if policy denies most of a candidate set, fetching more candidates lets authorized content deeper in the index surface. The problem was the initial fetch asked the vector database for exactly top_k candidates. Deny even one of them and you have fewer than top_k allowed results, which triggers a refill: a full second query, on a fresh database connection, roughly 110ms. In any realistic corpus where policies deny something, which is the entire point of the product, nearly every request paid for two searches.

The fix is the standard pattern for permission-filtered retrieval: over-fetch. We now request three times top_k up front and trim after evaluation, walking candidates in rank order and cutting after the top_k-th allowed result so the response, the counts, and the outcome are identical to what fetch-then-refill would have produced. Refill still exists for heavily-denied corpora; it just stopped being the common path.

Our first version of that fix shipped without the trim, and our live end-to-end suite caught it within one run: a principal whose retrievals had always been fully allowed started coming back "partial," because the wider fetch pulled in low-ranked candidates from other departments that were correctly denied. The assertion that failed was comparing outcome semantics across a 24-cell access matrix. Tests that encode meaning, not just status codes, are the reason we could make an aggressive performance change to the enforcement path and trust it.

The real number, and its conditions

After the three fixes, on the same methodology:

That is the headline run, and every run is published, not just this one. We re-ran the identical benchmark four more times on the same shared development machine and put all five in one artifact: p95 ranged from 13.6 to 26.2ms, and p50 sat between 10 and 16ms in every run we recorded. Most runs on this shared machine had at least one multi-hundred-millisecond stall (worst single requests ranged from 26 to 895ms across the five runs); in four of five runs the p95 absorbed it. The one run that exceeded 25ms carried a 307ms stall, about five times its own p99, which is host contention, not software. The headline figure is deliberately the conservative middle of that set, the exact median of the five p95s, not the best of them. The benchmark ran with debug logging enabled, which if anything inflates our numbers relative to production settings.

What the overhead includes: principal load, metadata resolution against the chunk registry, policy evaluation, the audit record and usage accounting writes, and response assembly. What it excludes: the vector query itself, and server-side query embedding (you can supply your own vectors; if we embed for you, that's an embedding-provider call, not policy work). HTTP middleware and JSON serialization add roughly 30ms at p50 and 45ms at p95 on our dev setup; the full series (46.5ms p50 / 65.4ms p95 in the headline run) is in the artifact too, because quoting only the flattering series is how we got into this situation.

What's still slow, and it's ours

The honest end-to-end picture: the dominant cost in a Gateco retrieval today is not the policy layer, it's the connector leg, which measured roughly 126ms p50 in the headline run. Most of that is not the vector query. It's opening a fresh database connection for every search, including driver-level type introspection for the vector extension, which accounts for something like 110ms of the leg. That is our code, it's filed, and the fix is a bounded per-DSN connection pool. Worth stating plainly: this benchmark measures the cold-connection case against a local database. Warm same-region query times are a different, smaller number that we have not benchmarked per connector, and until we have, we won't publish one. Until the pool ships, the accurate framing of our product is that it adds little on top of retrieval, not that it makes retrieval fast.

What's now permanent

Three things outlive this exercise. The benchmark is a script anyone can run against their own deployment, and everything is public: the script at gateco.ai/benchmarks/policy_overhead_bench.py, the headline run at gateco.ai/benchmarks/policy-overhead-bench-2026-08-16.json, and all five recorded runs at gateco.ai/benchmarks/policy-overhead-bench-all-runs.json. Run it yourself. Every production retrieval now records the total-versus-connector split, so the overhead figure is continuously verifiable from real traffic rather than from a lab run we did once. And the website copy was corrected: the overhead claim now cites the measured figures with their conditions, the end-to-end figures we had never measured were removed or labeled as unmeasured, and a CI check now blocks the phrases that started this from coming back.

The uncomfortable question that started this took one sentence to ask and a day to answer honestly. The claim survived, barely, at 21ms against a 25ms promise. The three bugs it flushed out were worth more than the number.