01Who calls this — every product team, or a few?
Every team: the gateway is the only door to model providers — one API key per team, one policy per use case, so failover, quotas, and redaction exist exactly once instead of forty times.
Free full guide
LLM Inference Gateway
Design an LLM inference gateway that sits between internal product teams and multiple model providers/backends, handling routing, streaming, quotas, and safety controls at company scale.
00
Practice checkpoints
The interview rhythm stays compact, so the page can spend attention on the actual design decisions.
- 01Clarify scope
- 02Requirements + scale
- 03API + data model
- 04Draw architecture
- 05Deep dive
- 06Trade-off decision
01
Requirements that shape the design
Do not only state requirements. Ask for them. Each card pairs the design constraint with a clarification question you can say out loud before drawing the architecture.
Functional requirements
02A provider starts failing — whose problem is it?
The gateway’s: each use case declares a fallback chain of schema-compatible models, health-checked per provider, and responses carry model_used and fallback_reason so callers can tell.
03Do responses stream through, or does the gateway hold them?
Yes — tokens pass through the gateway the moment the provider emits them, never buffered; the gateway meters tokens mid-stream so quota and cost see streaming traffic too.
04Are limits counted in requests or in tokens?
Tokens: a request can cost 100 tokens or 100,000, so each team gets a tokens-per-minute rate and a monthly dollar budget — request counts are blind to the actual cost unit.
05The same prompt arrives a thousand times — generate a thousand times?
No: an exact-match then semantic cache answers repeats, keyed by tenant scope, model version, and prompt-template version — personalized or user-specific traffic never enters it.
06The provider ships a new model version — who upgrades, and when?
Per-team policy: pin a version for output stability, or auto-upgrade behind an eval gate and canary — and pinned teams get a countdown the day the provider deprecates their model.
Out of scopeTraining or fine-tuning models (serving traffic only) · Building the GPU serving stack itself (that is the provider’s side) · End-user identity and sessions — callers are internal services
Non-functional requirements
01How much latency may the gateway itself add?
The gateway adds ~10-20 ms on the request path — first-token latency is the product metric, and the gateway is never allowed to become the reason it doubles.
02How fast must a dying provider be detected?
A dying provider is detected within seconds: an error-rate window per provider and model trips a circuit breaker, and the window length is chosen as a user-pain budget, not a config default.
03What gets logged from prompts and responses?
Metadata is always logged (tokens, model, cost, latency, trace id); prompt and response content only after the PII (personally identifiable information) redaction hook runs — raw prompts in logs turn the gateway into a compliance incident.
04Can a burst of parallel requests overshoot a team’s budget?
Budget checks are reserve-then-settle: max_tokens is reserved before the call and settled to actual usage after — read-then-write checks let a hundred concurrent requests each see money that is already spent.
05Spend spikes at 2 a.m. — how fast do we know who and why?
Every call lands in a usage ledger with team, use case, model, tokens, cost, cache and fallback flags — attribution is queryable in near real time, not reconstructed from provider invoices at month end.
Keep asking — the interview is a conversation
Real interviews probe far more than a tidy list. These are the scope questions that separate candidates who interrogate the problem from those who recite it.
- Which traffic goes through the gateway: interactive chat, background batch, embeddings — and which of those must stream?
- Is a team’s budget a hard cap that rejects at the limit or a soft cap that alerts — and who can override it mid-incident?
- Which teams need output stability enough to pin a model version, and what runway do they get when that version is deprecated?
- What is the first-token latency target, and how many milliseconds of it may the gateway itself spend?
- What may be persisted from prompts and responses — raw content, redacted content, or metadata only?
02
Numbers that force architecture decisions
Treat every estimate as a pressure that justifies a component: cache, queue, partition, replica, worker pool, or fallback path.
Cost attribution — tokens, not requests
One team: 200,000 requests/day, ~3,000 input + 800 output tokens each, at $3 per 1M input and $15 per 1M output tokens600M input × $3 + 160M output × $15 = $1,800 + $2,400 = $4,200/day ≈ $126K/month
A per-request rate limit cannot see this bill at all — quotas and budgets must be denominated in tokens, input and output priced separately.
Failover detection budget
At the ~100 requests/s workday peak (from the 2M/day org volume) the primary provider carries most traffic; the breaker uses a 10 s sliding window with a 20% error thresholdtotal outage → errors reach 20% of the ~1,000-call window after ~2 s, so ≈ 200 requests fail before the breaker opens
Window length IS the user-pain budget: shorter trips faster but flaps on one-second blips — pick it as a product number and pair it with half-open probes.
Streaming vs buffered first token
An 800-token answer generated at ~60 tokens/s; a buffering gateway holds the full response before forwardingbuffered first byte: 800 ÷ 60 ≈ 13 s; passthrough first token: ~0.5-1 s → roughly 15-25× worse perceived latency
Every synchronous hop added to the stream path is paid on all requests — the stream must be inspected in motion, never parked.
Semantic cache economics
Org-wide 2M requests/day, 15% combined exact + semantic hit rate, ~$0.021 provider cost per request (from the token math above)300,000 hits × $0.021 ≈ $6,300/day ≈ $190K/month saved; a hit answers in ~50 ms instead of ~13 s
Real money and a huge latency win — but every hit is a chance to serve the wrong answer, so keys must include tenant, model version, and template version.
Queue depth in a brownout
A brownout halves the primary provider’s throughput for 5 minutes at the ~100 requests/s peak: 100 arriving, 50 served, nothing shedbacklog grows (100 - 50) × 300 s = 15,000 requests; at ~20 requests/s spare capacity after recovery, draining takes another ~12 minutes
Queueing everything turns 5 minutes of brownout into ~18 minutes of degradation and 15K open connections — shed batch traffic and fail interactive traffic over instead.
Decision example
The numbers
Two million requests a day across forty teams at roughly two cents each is about $42,000 a day of provider spend — running on providers that each brown out for a few minutes some week of the month.
My choice
I would build one gateway path: authenticate the team, reserve max_tokens against its budget, check the exact-match then semantic cache, route by the use case’s model chain with a per-provider error-rate breaker, and stream provider tokens straight through while metering them mid-stream. Usage settles into a per-team ledger; prompt content touches logs only after the redaction hook; model versions are per-team policy — pinned with a deprecation countdown, or auto-upgraded behind an eval canary.
Avoid
What I would NOT do: let teams call providers directly — that is forty implementations of failover and zero shared attribution. I would not count requests for quota: a 100-token ping and a 100,000-token agent loop are not the same spend. And I would not buffer streams to inspect them — a gateway that waits for the full response turns a 1-second first token into 13 seconds of silence.
Change if
If there is one team, one model, and four-figure monthly spend, a thin shared client library with retries is the honest answer — the gateway earns its complexity only when teams, providers, or spend multiply.
03
Architecture path
Draw the write path and the read path as separate lanes — they carry different traffic and justify different components.
The hot path is auth → budget reserve → cache → route → stream; ledger settlement and health bookkeeping sit off it. A breaker trip changes only the router’s choice — callers keep the same contract and simply see model_used change.
The gateway counts tokens as they flow and settles the ledger from reserved to actual at stream end — quota and cost see streaming traffic exactly like blocking calls.
The fallback must be schema-compatible with the primary, and the response says which model answered — silent substitution breaks every team that depends on output shape.
04
API and data model
Before optimizing, make the contract inspectable: endpoints, entities, ownership, retries, and state.
POST/v1/responses
req{ use_case, messages, max_tokens, stream: true, output_schema? }
resSSE token stream; trailers carry model_used, token counts, cost, cache/fallback flags
Team identity comes from the API key, never the request body — one contract in front of every provider, so switching providers never touches product code.
GET/v1/usage (team_id, window=month)
res{ tokens, cost_usd, by_model, by_use_case }
Cost attribution as a product feature — team leads and finance read this, not provider invoices.
PUTinternal: policy(team_id, changes)
reshot-reloaded policy: pins, budgets, model chain, kill switch
Turning off a runaway team is a config write, not an emergency deploy.
Core entities
TeamPolicyteam_id (PK) · use_case · model_chain (primary → fallbacks) · version_policy: pinned/auto · tokens_per_min · monthly_budget_usd · redaction_profile
Hot-reloadable — a budget change or a kill switch must not wait for a deploy.
UsageLedgerEntryrequest_id (PK) · team_id · model_used · input_tokens · output_tokens · cost_usd · cache_hit · fallback_reason?
The attribution source of truth — written as a reservation pre-call, settled to actuals post-call.
ProviderHealthprovider + model · error_rate (sliding window) · p95 latency · breaker: closed/open/half-open
What the router reads before every dispatch; half-open probes decide when traffic is allowed back.
CacheEntryembedding key · tenant_scope · model + template version · response · ttl
A hit that ignores any one of these keys is how a stale or cross-tenant answer ships.
05
Deep dive directions
Pick one lane for the final third of the interview. Each lane gives you the topic, the interviewer question it should answer, and the failure mode to avoid.
FocusMid-stream provider death
AskA stream dies at token 400 of 800 with a provider 500. What does the client see, and can you fail over an in-flight generation?
AvoidSilently regenerating on provider B and splicing tokens — the client already rendered half an answer, and the new model will not produce the same continuation; resume is a restart and must be explicit.
FocusBudget gone by Tuesday
AskOne team’s runaway agent loop burned the org’s monthly budget in two days. Walk the controls that should have existed, in the order they should have fired.
AvoidA monthly invoice as the only control — per-call token metering, per-team caps, and alerts on burn RATE (dollars per hour against baseline) catch this on day one, not day thirty.
FocusThe cache lies
AskA user changes one critical word in the prompt and still gets the cached answer to the OLD prompt. Where did the semantic cache go wrong, and what bounds it?
AvoidSimilarity threshold as the whole safety story — negation flips meaning at 0.98 cosine similarity; keys need tenant, model, and template version, personalized traffic stays out, and hits carry a TTL.
FocusThe model retires
AskThe provider announces your pinned model dies in 90 days. Whose problem is that, and what does the gateway’s upgrade path look like?
AvoidAuto-upgrading everyone silently — teams pinned for output stability get an eval gate on their own golden prompts, a canary slice, and a sign-off, with the countdown surfaced from day one.
FocusThe moderation tax
AskSafety adds a synchronous moderation call before generation and first-token latency doubles. How do you keep the guardrail AND the latency budget?
AvoidSerializing every check in front of the provider call — input checks run in parallel with routing, and output moderation scans the stream in rolling windows with a cut-off, instead of buffering the whole answer.
Ready to practice?
Talk through LLM Inference Gateway out loud and get AI scoring on the explanation.