Moderation Service

01What gets screened, and does the user wait for it?

Every submission passes a synchronous fast path at post time — a perceptual-hash match against known-bad media plus a cheap classifier; heavier ML scoring happens asynchronously after the content is live.

02What is allowed to block posting, versus taken down after the fact?

Only the fast path blocks at submit: a hash hit or a near-certain classifier verdict rejects immediately; everything else posts right away and the async tier can retract it minutes later.

03The model is unsure — who decides?

Two per-category confidence thresholds split the async score: above the upper one auto-remove, below the lower one auto-allow, and the band in between routes to a human review queue — the thresholds are the human-workload dial.

04A user says the takedown was wrong — then what?

An appeals flow routes the case to a different reviewer who sees the original evidence and the policy version it was judged under; overturns restore the content and feed back into thresholds and training data.

05Do user reports actually do anything?

User reports raise a post’s review priority alongside predicted reach and violation severity, so the queue is ordered by expected harm, not arrival time.

06Can we prove, later, why something was removed?

Every decision — automated or human — is logged with the model score, evidence, and policy version, so appeals and regulators can replay exactly what was known at decision time.

01How much latency can moderation add to posting?

The sync fast path adds at most about 100 ms to posting, enforced with a hard timeout that fails open to the async tier — moderation must never become the posting-latency story.

02Is a false positive as bad as a false negative?

Error cost is asymmetric per category: severe-harm categories tolerate false positives to avoid a single miss, while borderline speech tolerates false negatives — so thresholds are set per category, never globally — and quality is tracked platform-wide as prevalence (the share of views that hit violating content) and the proactive rate (violations caught before any user report).

03How fast must harmful content come down?

Time-to-action scales with exposure: content predicted to go viral is reviewed within minutes, tail content within a day — harm is views times time, not just count of bad posts.

04Adversaries adapt — is that a design input?

The pipeline expects adversaries to adapt: re-encoded media and obfuscated text are normal inputs, so detection is layered (hash, classifier, reports) with no single point that evasion fully defeats.

05Are the human reviewers part of the system design?

Reviewer wellbeing is a system requirement — media blurred by default, exposure limits, category rotation — because burned-out reviewers make worse decisions, and decision quality is a system metric.