01What is the crawl FOR — search index, LLM training data, archival? It sets size and freshness.
Fetch pages starting from seeds and store raw content for downstream use — the consumer defines what "done" means.
Free full guide
Web Crawler
Design a web-scale crawler that politely fetches and indexes billions of pages, deduplicates content, and keeps the corpus fresh.
00
Practice checkpoints
The interview rhythm stays compact, so the page can spend attention on the actual design decisions.
- 01Clarify scope
- 02Requirements + scale
- 03API + data model
- 04Draw architecture
- 05Deep dive
- 06Trade-off decision
01
Requirements that shape the design
Do not only state requirements. Ask for them. Each card pairs the design constraint with a clarification question you can say out loud before drawing the architecture.
Functional requirements
02How does the crawl grow — do extracted links feed back in?
Every fetched page is parsed for links, which are filtered and fed back into the frontier — the crawl sustains itself from the seeds.
03Politeness: requirement or nice-to-have?
A requirement: obey robots.txt (including crawl-delay), identify honestly via user-agent, and never overload any single host.
04Can operators inject seeds and priority URLs mid-crawl?
Yes: the frontier accepts operator enqueues at elevated priority ahead of discovered links - a crawl is steered, not fire-and-forget.
05What exactly do we store - raw bytes, parsed text, or both?
Both: raw content in blob storage plus extracted text and metadata - downstream reprocessing must never require refetching the web.
06How fresh must robots.txt be?
Cached per domain with a bounded TTL (hours): serving from a stale allow is a compliance risk, so expiry forces a refetch before that host is crawled again.
Out of scopeSearch ranking and indexing (the crawler produces the corpus, not the index) · JavaScript rendering of dynamic pages · Continuous re-crawling for freshness (single full crawl first)
Non-functional requirements
01How many pages, and how fast?
10B pages in under 5 days — that single line dictates fleet size and queue throughput.
02What protects the websites we crawl?
Per-host rate limiting honoring crawl-delay: all URLs of one host flow through one queue with a token bucket — a million queued URLs for one site drain slowly, never as a flood.
03What stops us fetching the same thing twice?
Two dedup layers: a URL-seen filter before enqueue, and a content-hash check after fetch (same page, different URL).
04A fetcher machine dies mid-crawl — then what?
Nothing is lost: URLs are leased, not deleted — an unacknowledged lease times out and the URL returns to the queue; failures retry with backoff into a dead-letter after N attempts.
05What about infinite URL spaces?
Depth caps, per-domain page budgets, and URL-pattern filters keep calendar pages and ad farms from consuming the crawl.
Keep asking — the interview is a conversation
Real interviews probe far more than a tidy list. These are the scope questions that separate candidates who interrogate the problem from those who recite it.
- What is the corpus for — search, LLM training, archive? It sets scope and freshness.
- How many pages and what deadline should I size for?
- Do we render JavaScript, or fetch raw HTML only?
- How aggressive can politeness be — strictly robots.txt, or also adaptive backoff on 429/503?
- Is one full crawl enough, or does the corpus need continuous refresh?
02
Numbers that force architecture decisions
Treat every estimate as a pressure that justifies a component: cache, queue, partition, replica, worker pool, or fallback path.
Required fetch rate
10B pages ÷ 5 days10,000,000,000 ÷ 432,000 s ≈ 23K pages/s sustained
This is a fleet, not a server — and the frontier must hand out 23K polite URLs per second.
Fleet size
~2 s average fetch latency per page, so ~0.5 pages/s per connection · ~4K useful concurrent connections per box23K ÷ (4,000 × 0.5) ≈ 12 machines sustained — provision ~2× for retries and slow hosts
A couple dozen fetchers meet the deadline; the bottleneck is politeness, not compute.
URL-seen memory
10B URLs in a Bloom filter at 1% false positives (~10 bits each)10B × 10 b ≈ 12 GB — vs ~400 GB for exact strings
The Bloom answer "definitely new" is what matters: a false positive skips one URL, a false negative never happens — so nothing is crawled twice.
Storage for the corpus
10B pages × ~100 KB of stored HTML — the multi-MB quoted page weight is mostly images and scripts this crawler never fetches≈ 1 PB raw
Blob storage with compression; metadata (hashes, URLs) stays queryable in a separate store.
DNS pressure
23K fetches/s each needing resolutionwithout caching: 23K lookups/s · with per-fetcher DNS cache: ~1 per new host
An in-fleet DNS cache is mandatory — public resolvers would rate-limit the crawl to death.
Decision example
The numbers
Twenty-three thousand pages a second for five days straight — but the hard constraint is the opposite direction: no single website may ever feel more than a trickle.
My choice
I would split the frontier in two stages: front queues by priority, back queues strictly one per host, with a token bucket per host honoring crawl-delay. Fetchers lease URLs (never delete), ack on success, and let lease timeouts recover crashes. Before enqueue, URLs pass a Bloom-filter seen-check; after fetch, a content hash catches the same page under different URLs. Robots rules cache per domain with a TTL, and DNS gets its own cache inside the fleet.
Avoid
What I would NOT do: use one global priority queue — the moment a big site dumps a million URLs, fetchers hammer that host and the crawl becomes a DDoS. I also would not track seen URLs in an exact database table: 400 GB of strings and a lookup per enqueue, when a 12 GB Bloom filter answers "definitely new" for free — and its rare false positive merely skips a URL, which is the safe direction.
Change if
If the corpus needs continuous freshness instead of one crawl, the frontier gains a re-crawl scheduler: pages re-enter by change frequency and importance, and the seen-filter switches to a structure that supports aging out.
03
Architecture path
Draw the write path and the read path as separate lanes — they carry different traffic and justify different components.
Extracted links loop from the parser back into the frontier — the crawl feeds itself. Politeness lives in the frontier’s per-host back queues, not in fetcher goodwill.
A URL is only handed out when its host has a token; the lease returns to the queue on crash, retries back off, and repeated failures land in a dead-letter queue.
Every page yields links; filters drop traps and junk, the Bloom filter drops everything already seen, and the survivors re-enter the frontier.
04
API and data model
Before optimizing, make the contract inspectable: endpoints, entities, ownership, retries, and state.
QUEUEfrontier.enqueue(url, depth)
resaccepted · dropped (seen / filtered / over budget)
Internal contract, not REST — a crawler has no public API. Enqueue passes the URL-seen filter and pattern filters first.
QUEUEfrontier.lease(fetcher_id) → CrawlTask
restask with lease TTL · ack(task) on success · nack → retry with backoff
Per-host back-queues enforce politeness: a fetcher only receives a URL when that host’s token bucket has a token.
GEThttps://{host}/robots.txt (external)
resrules cached in DomainState with TTL
The one external contract: obey Disallow and crawl-delay, and send an honest User-Agent.
Core entities
CrawlTaskurl · domain · depth · status: queued/leased/done/failed · retries
The unit of work; leased (not deleted) while a fetcher works on it, so crashes self-heal.
DomainStatedomain (PK) · robots_rules · crawl_delay · last_fetch_at · pages_crawled
Robots rules are cached per domain with a TTL; the token bucket lives here.
Pageurl · content_hash (SHA-256) · fetched_at · blob_ref
content_hash powers the second dedup layer — identical content under different URLs stores once.
05
Deep dive directions
Pick one lane for the final third of the interview. Each lane gives you the topic, the interviewer question it should answer, and the failure mode to avoid.
FocusA million URLs, one small site
AskThe frontier holds a million URLs for one modest website. What guarantees it never gets hammered?
AvoidRate-limiting inside fetchers — politeness must be structural, in the per-host queue, or a scaled-out fleet breaks it.
FocusHave I seen this URL
AskTen billion URLs — how do you answer "seen before?" per enqueue, in memory, and what does a Bloom false positive cost here?
AvoidTreating the false positive as an error — skipping one URL is the designed cost; crawling twice is the failure.
FocusSame page, different URL
AskMirrors, tracking parameters, and www/non-www all serve identical content. Where does the second dedup layer sit, and on what key?
AvoidURL normalization alone — only a content hash catches true duplicates across hosts.
FocusThe infinite calendar
AskA site generates a valid "next month" link forever. What bounds the crawl, and how do you detect the trap?
AvoidRelying on depth alone — per-domain budgets and URL-pattern heuristics have to back it up.
FocusA fetcher dies holding 4,000 URLs
AskOne machine crashes mid-fetch. What happens to its in-flight work, and what does the recovery cost?
AvoidDeleting a URL from the queue at hand-out time — lease with a timeout, ack on completion.
Ready to practice?
Talk through Web Crawler out loud and get AI scoring on the explanation.