01Does the agent answer everything, or triage first?
Triage first: every inbound message is classified for intent — answerable from knowledge, needs account data, or belongs with a human immediately (angry customer, legal threat, churn risk).
Free full guide
AI Customer Support Agent
Design an AI customer support agent for a SaaS product.
00
Practice checkpoints
The interview rhythm stays compact, so the page can spend attention on the actual design decisions.
- 01Clarify scope
- 02Requirements + scale
- 03API + data model
- 04Draw architecture
- 05Deep dive
- 06Trade-off decision
01
Requirements that shape the design
Do not only state requirements. Ask for them. Each card pairs the design constraint with a clarification question you can say out loud before drawing the architecture.
Functional requirements
02What may the agent actually DO to an account?
Only approved tools with typed inputs — check status, update settings, issue a bounded refund. Risky or irreversible actions require explicit confirmation or a human.
03When must a human take over, and what do they receive?
On low confidence, repeated failure, or customer request: the human gets a handoff packet — conversation, retrieved sources, attempted actions, and the agent’s own uncertainty — never a cold start.
04Does the agent remember earlier turns - and earlier conversations?
Within a conversation, always (context window plus summarization for long threads). Across conversations, only account facts - never prior chat content, per the retention policy.
05One language or many?
The pipeline is language-agnostic, but evaluation is not: golden sets are per locale, because answer quality does not port across languages.
06Can a customer demand a human at any point?
Always - 'talk to a human' is one turn away at every step; hiding the exit inflates deflection metrics (the share of conversations that never reach a human) while destroying trust.
Out of scopeVoice channel and real-time speech · Training custom foundation models · Sales and marketing conversations (support only)
Non-functional requirements
01What is the worst failure mode — being slow, or being wrong?
Being wrong: an invented policy or an unauthorized refund costs more than any latency. Grounding, citation, and action gating outrank fluency.
02What data can the agent see while answering?
Only what this customer may see: retrieval is permission-filtered per tenant and per plan; internal runbooks and other tenants’ data are excluded at the index query, not post-filtered.
03How fast should answers feel?
First token in about 2 seconds, full grounded answer well under 30 — streaming makes waiting feel like typing.
04When something goes wrong, can we reconstruct why?
Yes: every turn is traced — query, retrieved chunks and scores, prompt, tool calls with inputs/outputs, and the escalation decision. Failures replay into the eval set.
05How do we know the agent is getting worse?
A standing eval suite — golden conversations with rubrics, judged pairwise, calibrated by humans — runs on every model or prompt change; deflection rate alone is not quality.
Keep asking — the interview is a conversation
Real interviews probe far more than a tidy list. These are the scope questions that separate candidates who interrogate the problem from those who recite it.
- Which actions may the agent take autonomously, and which always need a human?
- What knowledge is in scope — public docs only, or internal runbooks with permission tiers?
- What is the escalation contract — how fast must a human respond after handoff?
- What does success mean — deflection rate, resolution quality, CSAT, or all three?
- What are the data retention and PII rules for conversation logs?
02
Numbers that force architecture decisions
Treat every estimate as a pressure that justifies a component: cache, queue, partition, replica, worker pool, or fallback path.
Deflection economics
10K conversations/day, agent fully resolves ~60%6,000 × ~15 min ≈ 1,500 agent-hours/day ≈ 190 support seats absorbed
The business case lives or dies on resolution QUALITY — which is why the eval suite is a requirement, not tooling.
Context budget per turn
~8K tokens: system + policies (2K) + retrieved chunks (4K) + conversation window (2K)retrieval must rank well enough that 4K tokens carry the answer
Retrieval precision is the real quality lever — a bigger window is a cost increase, not a fix.
Latency decomposition
p95 targets: retrieval 300 ms + rerank 200 ms + first token 1.5 s≈ 2 s to first streamed token
Each stage gets its own budget and its own monitoring — "the AI is slow" is not a diagnosis.
Eval suite cost
500 golden conversations × 3 judged variants per release1,500 judged runs ≈ minutes of wall-clock in parallel, dollars per release
Continuous evaluation costs less than one mishandled enterprise ticket.
Escalation load
~40% of 10K conversations escalate4,000/day routed to humans WITH handoff packets
Handoff quality determines whether humans trust the agent — a bad packet doubles their work.
Decision example
The numbers
Ten thousand conversations a day, sixty percent fully resolved by the agent — the entire value rests on those resolutions being RIGHT, because one invented refund policy erases a month of savings.
My choice
I would build the agent as a gated pipeline: classify intent first, retrieve with the customer’s own permissions, answer only with citations, and put every side effect behind typed, allow-listed tools with risk tiers enforced outside the model. Confidence gates every step — below threshold, the customer gets a human plus a handoff packet carrying the conversation, sources, attempted actions, and the agent’s uncertainty. Every turn is traced and replayable, and failures feed a golden eval set judged by rubric and pairwise comparison, calibrated by humans.
Avoid
What I would NOT do: hand the model a database connection and a system prompt that says "be helpful" — free-form access is how an agent refunds the wrong customer. I also would not measure quality by deflection rate alone: an agent that confidently deflects with wrong answers scores perfectly while destroying trust. And exact-match tests cannot judge support conversations — "did it resolve correctly" is a preference judgment, which is why the eval set stores good AND bad reference outputs with rubrics.
Change if
If the product goes multi-region with data residency rules, retrieval indexes and conversation logs must shard per region, and the eval suite gains per-locale golden sets — quality is not portable across languages.
03
Architecture path
Draw the write path and the read path as separate lanes — they carry different traffic and justify different components.
Everything the model does passes a gate: retrieval is permission-scoped, actions go through the tool gateway, and low confidence exits to a human with a full handoff packet. Traces record every hop.
Intent classification routes before any generation; retrieval carries the customer’s permissions into the index query; the answer cites its sources or it does not ship.
The model proposes; the gateway disposes. Low tiers execute, medium tiers ask the customer to confirm, high tiers require a human — and every call lands in the trace.
04
API and data model
Before optimizing, make the contract inspectable: endpoints, entities, ownership, retries, and state.
POST/support/messages
req{ conversation_id?, text }
res200 streamed answer with citations · or { escalated: true, ticket_id }
Customer identity comes from the session — the agent’s permissions are the customer’s permissions, never broader.
POST/support/{conversation_id}/escalate
res201 { ticket_id } with the handoff packet attached
Fires on low confidence, repeated failure, or explicit customer request — escalation is a feature, not a failure.
POSTinternal: tools.execute(tool, input, risk_tier)
resresult · blocked (needs confirmation) · denied (policy)
The only path from model output to real side effects: typed inputs, allow-listed tools, risk tiers enforced outside the model.
Core entities
Conversationconversation_id (PK) · customer_id · channel · state: active/escalated/resolved
Turnturn_id (PK) · conversation_id · role · content · trace_ref
trace_ref links to the full replayable trace: retrieval, prompt, tool calls, decisions.
ToolActionaction_id (PK) · conversation_id · tool · input · result · risk_tier · confirmed_by
Risky tiers record who confirmed — the agent, the customer, or a human agent.
HandoffPacketconversation_id · summary · retrieved_sources · attempted_actions · uncertainty_notes
What the human receives at escalation — the difference between a takeover and a restart.
05
Deep dive directions
Pick one lane for the final third of the interview. Each lane gives you the topic, the interviewer question it should answer, and the failure mode to avoid.
FocusThe agent promises a refund it should not
AskWhat are the layered defenses between "the model wants to refund $500" and money actually moving?
AvoidTrusting the system prompt as the control — policy lives in the tool gateway, outside the model.
FocusTenant data stays home
AskTwo companies use this product. Walk retrieval for a question whose best-matching chunk belongs to the OTHER tenant.
AvoidPost-filtering retrieved chunks — the permission filter must be inside the index query itself.
FocusThe handoff moment
AskConfidence drops mid-conversation. What exactly does the human agent see, and what happens to the customer’s flow?
AvoidEscalating with just a transcript — sources, attempted actions, and uncertainty notes are what save the human from restarting.
FocusIs it getting worse?
AskA prompt tweak ships. How do you know support quality did not silently regress — before customers tell you?
AvoidExact-match tests or deflection rate as the metric — support answers need rubric plus pairwise judgment, human-calibrated.
FocusWho saw what
AskA customer disputes an action the agent took last month. Reconstruct the conversation: what is stored, for how long, and who may read it?
AvoidStoring raw conversations forever with PII intact — retention and redaction are requirements, not cleanup.
Ready to practice?
Talk through AI Customer Support Agent out loud and get AI scoring on the explanation.