2σ Anomaly Detection at 2 AM: Catching Campaign Drops Before You Wake Up

Every night at 2:00 AM, a scheduled job scans every ad campaign's metrics and compares them against their own historical distributions using a statistical threshold: two standard deviations, or 2σ. A campaign whose impressions fall 67% from a mean of 301 is not “looking low” to a human — it is 2σ+ off, and it becomes an anomaly event with a score attached. You wake up to a diagnosis, not to a surprise.

Three layers: detection, reasoning, persistence

The system is deliberately three-layered. Layer one is pure statistics: threshold-based detection over historical means and deviations — no heuristics, no “feels off.” Layer two is the LLM reasoning pass that turns the statistical signal into a readable causal story. Layer three persists the whole thing — the event, the score, and the resulting evidence chain — so the discussion is never lost and the same anomaly can be reviewed later or used to train judgment. The statistics decide that something happened; the model explains it; the record keeps both honest.

The dedup rule: same campaign, same metric, once a day

Without deduplication, a slow-moving metric gets re-flagged on every scan and the alert channel becomes noise. Our rule is simple: if the same campaign and metric already produced an anomaly event today, skip it. One event per campaign-metric pair per day. The signal stays strong precisely because it is rare enough to read.

The data pipeline is the lifeline

Anomaly detection is only as good as the data it reads, and we learned this the hard way. Early on, campaign_stats was written incidentally by another agent, usually after our 2:00 AM detection run. The detector was reading yesterday's numbers as today's — quiet, systematic staleness. We fixed it by giving the snapshot its own dedicated job: a scheduled task now generates the daily campaign_stats snapshot at 1:30 AM, and the detection run at 2:00 AM is guaranteed to see a complete set of yesterday's data, every single day.

Rule: detection never runs before the snapshot is guaranteed. Data completeness is scheduled, not hoped for.

How to build your own 2 AM sentry

1. Snapshot first, detect second. Make daily state capture a dedicated, scheduled job that runs before detection, and fail the detection run loudly if data is missing.
2. Use a statistical threshold, not a fixed number. Two standard deviations from each metric's own mean adapts to the metric; a hard-coded “-30%” is arbitrary.
3. Dedup same-day alerts. Same campaign + same metric, one event per day, or the alerts become noise nobody reads.
4. Carry context on the event. Store the historical mean, the deviation magnitude, and the anomaly score alongside every event, so no one has to re-derive the numbers to judge it.
5. An alert is not the endpoint. Wire the event to reasoning and to a human review step — that is where the anomaly becomes an action.

The math does the watching; the pipeline does the waiting; the human does the deciding. This is exactly the architecture our attribution and growth engine at iport runs, with the 2σ scan, evidence chains, and review loops you can schedule with a click.