Distributed Key-Value Store

01What is the client-facing surface?

Three operations and no more: get(key) returns the value — or several sibling values plus a causal context when versions conflict; put(key, context, value) writes with the context the client last read; delete(key) writes a tombstone. Values are small opaque blobs (roughly 1 KB) — no scans, no joins, no transactions.

02Which nodes own a key?

The key hashes onto a consistent-hash ring and its preference list is the next N distinct physical nodes clockwise; every machine appears as many vnodes (virtual nodes — small slices of the ring it owns), so ownership moves in slivers, not continents, when membership changes.

03How many replicas must confirm a write? A read?

Tunable per operation: the coordinator fans a write to the N home replicas and acks the client at W confirmations, fans a read and answers at R — choose R + W > N and the read set must overlap the write set, so a quorum read sees the latest quorum write. N=3, W=2, R=2 is the default rung.

04Two home replicas are unreachable — does the write fail?

No — sloppy quorum: the write lands on the first N healthy nodes walking the ring past the failures, so a dead node or a partition never blocks a write; the guarantee that weakens (read/write overlap) is named out loud, not hidden.

05Both sides of a partition wrote the same key — who wins?

Neither, automatically: vector clocks — (node, counter) pairs on every version — prove the writes are concurrent, the store keeps both as siblings, and the next read returns them with a context so the application merges (or last-write-wins applies as an explicit, named policy for key classes that opt in).

06A replica was down for an hour — how does it catch up?

Three repair channels, fastest first: hinted handoff drains the writes its stand-ins held for it, read repair fixes any key a client happens to read stale, and background Merkle anti-entropy sweeps the cold keys nobody read — every replica converges without an operator touching it.

01The network partitions — which side stops serving?

Always writable — the CAP choice (consistency, availability, partition tolerance: a network split forces you to pick which of the first two survives) is made for availability: both sides of a partition keep accepting writes, and consistency is repaired after the fact, not demanded during it.

02Can one feature get safer reads while another gets faster writes?

Consistency is a per-operation dial, not a system property: each read and write picks its (R, W) rung — R + W > N where the product needs read-your-writes, W=1 where write availability is everything — and every rung in use is a recorded product decision.

03What does a quorum read owe at p99?

Single-digit-millisecond p99: a quorum read completes at the R-th fastest of N parallel replicas, so one slow replica never sets the tail — redundancy already paid for durability doubles as tail-latency shielding.

04A node crashes one second after acking a write — is the write gone?

Durability floor stated in acks: the default W=2 means an acked write survives any single node loss; W=1 exists on the ladder but ships with its risk named — a crash after the ack can erase the write.

05What does adding the 101st node cost the other hundred?

Incremental scale: adding one node to a hundred moves about 1% of the data, drawn nearly evenly from every existing node thanks to vnodes — capacity grows by routine, not by migration project.