Measuring Gateco Policy Overhead: 16ms p50, 21ms p95

Correction (2026-08-21): this post originally stated the overhead claim without a published benchmark, and an end-to-end figure we had never measured. It now cites the measured numbers only. The benchmark script and every recorded run are public at gateco.ai/benchmarks.

The latency question comes up in almost every evaluation: "If Gateco sits between my application and my vector DB, how much does it slow things down?" The measured answer is 16ms p50 and 21ms p95 policy overhead, benchmarked separately from the vector DB latency itself. We do not publish an end-to-end figure: the vector-store leg varies by deployment, and in our published run it was dominated by per-request connection setup (~126ms p50 against a local pgvector; a per-DSN connection pool is the filed fix). Here is what the overhead covers and what drives the variance.

What "policy overhead" means

Policy overhead is the latency Gateco adds to a retrieval request that does not include the vector DB round-trip. It covers: principal resolution (looking up the principal's attributes from the local principal store), policy evaluation (evaluating all matching policies against the principal and each returned chunk), audit record write (appending the decision to the audit log), and response assembly (filtering the result set and formatting the response). The vector DB latency is a separate dimension. Gateco does not change how fast your vector DB executes a query.

Principal resolution: the fast path

The most expensive part of policy evaluation is principal resolution: looking up a principal's groups, department, and attributes from the identity provider. Gateco solves this with a local principal store that is populated and updated by IDP sync. When a query arrives, principal resolution is a single local database read, not an outbound IDP API call, and the record is read live on every request rather than cached. Even for organizations with complex IDP graphs, per-query resolution is a few milliseconds; it is included in the measured overhead figures.

Policy evaluation: linear in policy count

Policy evaluation time is linear in the number of active policies and the number of returned chunks. In the published benchmark, evaluating a 5-rule policy over ~30 candidates took about a millisecond. The most expensive single operation in policy evaluation is a ReBAC lookup: checking whether a principal has a named relation to a resource requires a database read (with a 60-second result cache). As engineering estimates, not yet benchmarked: a first ReBAC lookup adds a few milliseconds and cached lookups are sub-millisecond. Classification ceiling checks and RBAC group membership checks are purely in-memory and add negligible overhead.

Variance by connector

The vector DB round-trip is the largest source of end-to-end latency variance. Vendor-typical warm-query figures, which we have not benchmarked per connector: Postgres-family connectors (pgvector, Supabase, Neon) in the same cloud region around 5-15ms; Pinecone's managed API 15-30ms; Qdrant's managed cloud 10-25ms; Azure AI Search and Google Vertex AI Search 20-50ms depending on index size and query complexity; OpenSearch 15-40ms. These exclude connection setup, which dominated the connector leg in our published run. The policy overhead figure is consistent across connectors because it is measured before and after the connector adapter call, not including the adapter's own latency.

What happens when policy evaluation is slow

The fail-closed guarantee means that if the policy engine encounters an error (a database connection failure, a timeout, an unexpected exception), the retrieval is denied and the decision is logged as `decision=error_deny`. There is no "policy evaluation timed out, allow anyway" path. The circuit breaker (5 errors in 30 seconds, half-open after 2 minutes) prevents a degraded policy engine from adding indefinite latency to every request. If the circuit is open, requests are denied immediately with a circuit-breaker reason in the audit log, which is a better outcome than a slow deny.

The latency data above comes from a public, reproducible benchmark (script and every recorded run at gateco.ai/benchmarks): repeated sequential request runs against a pgvector baseline with a 5-rule RBAC policy and 100 registered resources, measuring 16 ms p50 and 21 ms p95 policy-layer overhead. Every production retrieval records the same split (total vs connector latency) in its audit record, so you can compute the figure for your own deployment. If yours shows higher policy overhead, the most common causes are: excessive active policies (>50), large top_k values (>25), or ReBAC policies against un-cached relations. The docs performance page has the full methodology.