When a Self-Hosted Model Gateway Tripped Trust & Safety: Six Seats, Nine Seconds
The Nine Second Block
For eight months a self-hosted gateway on my internal orchestration node served every model behind one key and one model picker: a long-context model for reading large repos, a frontier model for refactors, an open-weight model for cheap background jobs. One endpoint, one dashboard, one place to see spend. It worked well enough that I stopped noticing what it was actually doing — presenting subscription OAuth tokens from software the vendor never authorised.
On a recent Friday morning, in the middle of a routine maintenance window, that arrangement ended in nine seconds. The gateway’s scheduled credential health check fired across all six connections at once, each returning:
403 This service has been disabled in this account
for violation of Terms of Service.
Please submit an appeal to continue using this product.The first connection failed immediately, and the last followed inside the same nine-second window. None of the six seats had been doing anything unusual; the sweep itself was the trigger, and it found a block that had already been applied upstream. The gateway’s own database flipped each connection to test_status = banned, error_code = 403 in a single pass, and the first failed chat in the orchestrator log landed two minutes later.
The failure mode that mattered was not the block. It was the ten minutes of confident misdiagnosis that followed, because the gateway surfaced an upstream policy decision as an HTTP 401 with invalid_api_key. Every client that consumed that error — the orchestrator, shell probes, the CLI — reported a credential problem. Those ten minutes went into rotating keys that were never the issue.
Why a Shared Token Is Not a Shared Key
An API key is a bearer credential issued to whoever holds it. An OAuth token is not. It is bound to a client identity: a client id, a consent record, and scopes granted to a specific application. When a third-party process presents that token, the request keeps the original client’s claims but loses the original client’s behaviour — headers, TLS fingerprint, request cadence.
That gap is precisely what Trust & Safety systems are built to detect, and the vendor’s terms said so in advance. The acceptable-access clause prohibited reaching the service through third-party software, and the FAQ drew the line finer still: scripting the official CLI directly was acceptable, while pooling seats or automating rotation across them was not supported.
Two properties of my setup made the pattern unmistakable:
- One egress. All six seats left through the same container on the same public address, so every request shared one network identity.
- One fingerprint. The gateway normalised headers and stripped client telemetry, so six seats looked like one client generating six times the traffic.
Nine seconds for six connections is not six human reviews. It is one rule matching one cluster. The seats had been configured for separate workflows — interactive work, CI jobs, staging probes — and I had collapsed their credentials into a single routing layer for convenience. That consolidation is the mistake, and it is independent of how many seats exist.
The Forensic Trail
The useful part of a post-mortem is the evidence an engineer can re-derive, not the conclusion. Three artifacts settled it within an hour.
| Artifact | What it proved |
|---|---|
| Gateway health endpoint | The gateway itself was healthy: HTTP 200, models listed, unrelated providers answering normally |
| A non-subscription model on the same key | Returned 200 with tool calls, so the credential and the routing layer were both fine |
| Every subscription model | Returned the identical 401, seconds apart, with the upstream 403 embedded in the body |
The third row is the one that matters. When one model fails, the model is broken. When every model on one provider fails identically while every other provider on the same key succeeds, the failure lives in the provider relationship, and no amount of key rotation will move it.
The gateway’s connection table confirmed the state, and its call log gave the shape of the event:
t+0s connectivity sweep 6 connections -> 403 policy block
t+9s last connection 6/6 disabled
t+28s orchestrator first failed chat request
t+96s connection table rows flipped to banned / 403🔍 The full isolation probe: four commands that separate credential, gateway and provider (27 lines)
# 1. Is the gateway itself alive?
curl -s -o /dev/null -w '%{http_code}\n' "$GW/api/health"
200
# 2. Does a model from an unrelated provider answer on the same key?
curl -s "$GW/v1/chat/completions" \
-H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
-d '{"model":"open-weight/unrelated","messages":[{"role":"user","content":"reply OK"}]}' \
| jq -r '.choices[0].finish_reason'
tool_calls
# 3. Do the subscription models fail identically?
for m in subscription/fast subscription/deep subscription/alt; do
curl -s "$GW/v1/chat/completions" \
-H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
-d "{\"model\":\"$m\",\"messages\":[{\"role\":\"user\",\"content\":\"ping\"}]}" \
| jq -r '.error.code // .choices[0].finish_reason'
done
invalid_api_key
invalid_api_key
invalid_api_key
# 4. What does the gateway's own connection table say?
sqlite3 gateway.db \
"select provider, test_status, error_code, count(*)
from connections group by 1,2,3;"
subscription|banned|403|6Row 2 succeeding while row 3 fails three times identically is the whole diagnosis: the key works, the gateway works, the provider relationship is closed.
Task Dispatch Instead of Routing
The lesson is not “use fewer seats”. The lesson is that a subscription is a seat, not an API. A seat grants a human, through the vendor’s own client, the right to work. The moment its token leaves that client, the seat becomes a shared credential, and enforcement treats it as one.
So the architecture changed shape. The gateway keeps serving models paid for by API key — that part was never the problem. Subscription work moved to task dispatch: the orchestrator runs isolated, single-task invocations of the vendor’s official CLI under a seat it is authorised to use, and the token never leaves the vendor’s binary. No pooled credentials, no persistent hijacked session, no rotation across seats.
orchestrator -> task dispatcher -> official CLI (owns the token) -> vendor
|
+- cwd, permission scope, timeout, tree-kill, no retriesFour properties of the dispatcher did more for safety than any prompt:
- Explicit working directory. Each task starts inside one repository, never in a user home directory or a filesystem root, because an agent with edit rights inherits the whole shell of the user running it.
- Closed stdin. Interactive prompts inside a headless run — a package manager asking to initialise, an SSH client asking to trust a host key — become errors instead of silent hangs. Without this, one forgotten
-yblocks the parent process forever. - Process-group kill on timeout. Killing the parent leaves grandchildren running; the dispatcher kills the whole group and returns a timeout code instead of a half-finished diff.
- No automatic retries. An inner agent loop multiplied by an outer retry policy produces hundreds of upstream calls in minutes — exactly the cadence the previous design was flagged for.
A minimal version of the execution core is small enough to audit in one screen:
proc = subprocess.Popen(
argv, cwd=repo, stdin=subprocess.DEVNULL, # no interactive hangs
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
start_new_session=True, # own process group
)
try:
out, err = proc.communicate(timeout=task_timeout)
except subprocess.TimeoutExpired:
os.killpg(proc.pid, signal.SIGKILL) # kill the whole tree
out, err = proc.communicate()
return {"status": "timeout", "detail": err[-2000:]}Context stopped travelling in prompts as well. The delegate reads repository instruction files and queries a self-hosted memory service over MCP, so a brief stays six lines instead of a transcript dump. Delegate writes are tagged status: draft, and final archiving stays with the orchestrator — a subagent that records its own failed attempts as fact poisons every later job.
One more fix belongs here, because it addresses the original ten wasted minutes: retained API-key traffic still passes through the gateway, and its error mapping now distinguishes classes instead of flattening them. An upstream policy block surfaces as 502 with upstream_policy_block metadata and fires an alert; authentication failures stay 401. A real credential problem and a vendor decision no longer look identical from the client side.
The Economics Still Work
Dropping OAuth reuse could have cost more than it saved, so the replacement had to clear a cost bar. It does, because the workload splits cleanly in two.
Routine orchestration — roughly nine turns in ten. High-frequency tool loops, retrieval, cron jobs, log triage and short code edits run on a curated open-weight plan: $10/month carrying a $100 monthly allowance on the fast model, with separate per-model pools beside it. Off-peak pricing is $0.15 per million input tokens and $0.60 per million output, so a 4k-in/500-out turn costs 4000 × 0.15/1e6 + 500 × 0.60/1e6 ≈ $0.0009 — just under a tenth of a cent. Two guardrails keep that allowance honest: per-model pools, so throttling one model does not stop the pipeline, and hard escalation rules — two failed attempts on the same error, three or more files touched, more than five exploratory steps, or any irreversible production operation moves the task off the cheap model before it burns the allowance.
Heavy engineering — the remaining tenth. Multi-file refactors, legacy archaeology and anything with a real build-and-test contract dispatch to an official CLI seat, where the vendor’s own client owns the credential and the allowance is the subscription’s, not a pooled one.
That split is the actual answer to keeping this cheap without crossing the boundary: pay for frequency with API-key models, pay for depth with authorised seats, and never let the two share a token.
Reflection
What I keep from this is that the gateway was genuinely good engineering solving a real problem — one interface for many providers, one place to see spend, failover that worked. What it lacked was a boundary. A credential issued to one client cannot be re-hosted by another and expected to keep its identity, and no amount of clean code changes what the enforcement layer sees: one address, six connections, one fingerprint.
The pooled pattern is now permanently retired. The seats that remain are used the way the vendor intends — through its client, one task at a time, at a cadence a human developer could plausibly produce — and the compliance work to move subscription access onto properly licensed tiers is in progress.
Task dispatch is slower to build and uglier to debug than a router that answers for everything. It is also the only version of this that survives an audit.
Want a 24/7 bot or automation pipeline that never drops a lead?
Don't let customer inquiries slip away at midnight or on weekends. I build custom Telegram/WhatsApp AI assistants and deterministic workflows that connect directly to your catalogue, CRM, and order fulfillment.
