Metric Monitoring and Alerting

Design the metrics pipeline behind a monitoring platform — the path from an SDK on thousands of hosts emitting numeric measurements, through streaming pre-aggregation, into durable time-series...

00

Practice checkpoints

The requirements are open as a taste. From the numbers onward, the full guide opens in the app.

  1. 01
    Clarify scope
  2. 02
    Requirements + scale
  3. 03
    API + data modelUnlocks in the app
  4. 04
    Draw architectureUnlocks in the app
  5. 05
    Deep diveUnlocks in the app
  6. 06
    Trade-off decisionUnlocks in the app
01

Requirements that shape the design

Do not only state requirements. Ask for them. Each card pairs the design constraint with a clarification question you can say out loud before drawing the architecture.

Functional requirements

01What leaves the host — every measurement, or a summary?

The SDK aggregates in-process and flushes once per interval (10 s): counters as running sums, latencies as histogram bucket counts — the wire carries series values, never per-request events.

02Which metric shapes must survive aggregation?

Counters, gauges, and histograms; percentiles are computed server-side by merging histogram buckets, because averaging per-host p99s produces a number that is simply wrong.

03How do people find and slice a metric?

By metric name plus label filters (service, region, endpoint) over any time range — a series is one unique combination of name and label values, and that identity drives storage and cost.

04How fresh does a dashboard have to be?

A value measured now is queryable within about 30 seconds — one SDK flush plus one aggregation window; the alert evaluator reads this same store and needs the same freshness.

05Can a team attach any label it wants?

Label keys are declared and label values are capped per metric; a value set that grows without bound (user id, request id) is rejected at ingest, because each new combination is a new permanent series.

06How far back can a dashboard look?

Years — at declining resolution: recent data at 10-second grain, older data as 1-minute and 1-hour rollups, so capacity planning works without paying raw-resolution storage forever.

Out of scopeAlert rule evaluation, dedup, and on-call paging — that is the sibling Monitoring and Alerting question; this pipeline feeds it fresh aggregates and stops there · Logs and distributed traces (different data shapes, different stores) — this system is numeric time series only · Anomaly detection and ML forecasting layered on top of the stored series

Non-functional requirements

01What ingest rate is the design point?

About 10 million points per second sustained, arriving as one batched flush per host per interval — and the SDK must never block or slow the application it measures.

02How fast must a 30-day panel render?

A 30-day panel renders in about a second — possible only because the query reads rollup tiers instead of scanning millions of raw points.

03What happens when ingestion outruns storage?

When storage falls behind, collectors buffer briefly and then shed the finest resolution first — backpressure never reaches the monitored application.

04Does monitoring survive the outage it is measuring?

Degradation is lossy but honest: missed intervals render as gaps, never interpolated — a monitoring system that invents data during an incident is worse than one that is down.

05Is series growth bounded, or can one team blow it up?

Cardinality is bounded by design: per-metric series caps enforced at the ingest gateway, because every unique label combination costs index space and memory for as long as it lives.

Keep asking — the interview is a conversation

Real interviews probe far more than a tidy list. These are the scope questions that separate candidates who interrogate the problem from those who recite it.

  • Is the alert evaluator reading this same store, and does it need data fresher than dashboards do?
  • Is losing the final in-memory interval from a crashed host acceptable, or does some metric here secretly need billing-grade durability?
  • Who governs labels — can any engineer ship a new label key, or does a new dimension go through review before it multiplies the series count?
  • What is the longest range anyone actually queries, and at what resolution do they genuinely need it?
  • Is this multi-tenant — can one team’s cardinality accident degrade every other team’s dashboards, or do we partition by namespace?
01

Unlock the full playbook for Metric Monitoring and Alerting

Numbers, architecture diagram, API and data model, deep dives, expected topics, self-check, whiteboard starter, and common mistakes unlock inside the app.

02

Numbers that force architecture decisions

Locked in the app

03

Architecture path

Locked in the app

04

API and data model

Locked in the app

05

Deep dive directions

Locked in the app

+Series

Practice the related series