AI Training Cluster

01What runs here — one team’s pretraining, or everyone’s everything?

A shared cluster: many teams submit training, fine-tuning, and eval jobs — each with tenant, priority tier, GPU count, expected duration, and a checkpoint policy the scheduler can read.

02A job asks for 512 GPUs and only 400 are free — start it partially?

Never: synchronous training moves in lockstep, so the gang launches all-or-nothing — the scheduler reserves freed GPUs toward the full set instead of leaking them to small jobs forever.

03Production needs capacity NOW and research is holding it — who wins?

Priority tiers: production preempts research, research preempts best-effort — and preemption is a contract: checkpoint signal, grace window, then reclaim, with the preempted job auto-requeued.

04Do teams get hard carve-outs, or one shared pool?

Fair-share quotas: each team owns a guaranteed share, but idle share is instantly backfillable by other teams’ preemptible jobs — capacity is loaned, never given away.

05A node dies four days into a two-week run — then what?

The whole gang halts (synchronous training cannot limp along), the scheduler swaps in a healthy node, and the job resumes from its last checkpoint — lost work is bounded by checkpoint cadence, not by luck.

06Where do training data and finished models live?

Datasets are staged onto node-local cache before launch so the input pipeline keeps GPUs fed; finished weights and checkpoints go to an artifact registry with lineage (job, data version, code version).

01What single number does the cluster owner wake up to?

Utilization of BUSY GPUs — allocated-but-idle counts as waste. At ~$2/GPU-hour, a 10,000-GPU cluster burns about $480K a day, so every stranded percent is real money.

02A job has waited six hours — can anyone say why?

Every scheduling decision is explainable — which constraint blocked placement (quota, gang size, topology, power envelope) — because an unexplainable queue destroys trust in shared infrastructure.

03How much work is one hardware failure allowed to destroy?

One failure destroys at most one checkpoint interval of work: cadence is derived from failure math (expected interruptions per day × GPU-hours per gap), not from a default config value.

04Can a 512-GPU job wait forever while small jobs keep landing?

The biggest job cannot starve behind a stream of small ones: gang reservations age, and the reserved-but-idle gap is backfilled only with preemptible work the scheduler can evict on time.

05What may telemetry record from other people’s training runs?

Telemetry records job metadata, GPU-hours, cost per tenant, and rack power/thermal headroom — never training data or model weights — because dense GPU racks hit the power envelope before they run out of space.