Quantifying the hidden engineering tax that noisy monitoring imposes on on-call teams — and what changes when alert volume drops instead of alert tooling multiplying.
Every engineering organization pays an alert tax. It doesn't show up on a P&L, it isn't itemized in a cloud bill, and no one requisitions budget for it — which is exactly why it survives so many rounds of cost-cutting that eliminate line items with actual invoices attached. It shows up instead as interrupted focus, as 2am pages that turn into 6am exhaustion, as senior engineers who quietly stop trusting the pager and start filtering it with their own judgment before they even open the laptop.
That last behavior is the tell. When experienced on-call engineers start pre-filtering alerts in their head — "that's probably nothing, I'll check in the morning" — the alerting system has already failed at its one job. The tax isn't paid in the alerts that get acted on. It's paid in the ones that get learned to be ignored, and the rare real incident that gets ignored along with them.
This paper does not introduce a new dollar-cost-per-alert statistic. Every specific number in it is either a figure already published elsewhere on this site from real customer engagements, or is explicitly marked Illustrative example where we're walking through example arithmetic rather than quoting a measured outcome.
Alert fatigue has a fairly mechanical origin: most monitoring systems alert on static thresholds — CPU above 80%, latency above 500ms, error rate above 1%. These thresholds are set once, usually during initial rollout, and rarely revisited as traffic patterns, deploy cadence, and service behavior evolve. A threshold tuned for Tuesday morning is wrong for Friday afternoon and wrong again for the 2am batch window.
The result is a flood of alerts that are technically accurate — the metric really did cross the line — but operationally meaningless, because the line was never a good proxy for "something is actually wrong." Engineers learn this quickly. What they learn next is to stop trusting the system that produces them, which is the actual mechanism of alert fatigue: not too many alerts in the abstract, but too many alerts that turned out not to matter, compounding into a system nobody fully believes.
Because "the cost of alert fatigue" isn't a number any vendor — including us — can hand you as a verified fact (it depends entirely on your team size, your incident rate, and your loaded engineering cost), the honest way to talk about it is to walk through the arithmetic explicitly, and label it for what it is.
Illustrative example — not a verified customer figure
Take a 12-engineer on-call rotation fielding 25 pages a week, of which roughly 90% resolve to "no action needed" (a ratio consistent with the false-positive rates reported before IntelliSense in the case referenced in Chapter 5). If each false page costs an engineer 12 minutes of context-switch and re-focus — a conservative estimate for anyone who has been pulled out of deep work by a phone buzzing at 2am — that's roughly 4.5 hours of engineering time consumed by noise every week, before any real incident work begins. Multiplied across a year, that's north of 230 hours: nearly six work-weeks of a single engineer's time, spent entirely on alerts that needed no action.
That arithmetic is illustrative — your ratios will differ — but the shape of it holds generally: false-positive rate compounds with team size and shift frequency to produce a tax that is invisible in any single week and enormous in aggregate. The fix is not more people on rotation. It's fewer false positives per real incident.
The reason static thresholds generate so much noise is structural, not a tuning failure: a single number can't represent "normal" for a service whose normal changes by time of day, day of week, and deploy state. IntelliSense, Applicare's anomaly detection engine, replaces the single threshold with a separate behavioral baseline for every entity in the environment — every service, every host, every database instance — learned from that entity's own history rather than a global rule.
A checkout service processing ten thousand requests a minute on a Friday afternoon has a different normal than the same service at 3am on a Tuesday. A per-entity baseline models both, without anyone writing a rule for either case. That is a direct, mechanical answer to the noise problem described in Chapter 2: the alert only fires when a specific entity deviates from its own established pattern, not when it crosses a number someone picked eighteen months ago.
| Alerting approach | What "normal" means | Adapts as traffic changes |
|---|---|---|
| Static threshold | One fixed number, set once | No — requires manual re-tuning |
| Aggregate baseline | Average across all instances of a metric | Slowly, and imprecisely |
| IntelliSense per-entity baseline | That specific entity's own recent history | Continuously, without configuration |
The 94% figure that anchors this paper isn't a lab benchmark. It's a measured result, published on our engineering blog, from comparing alert volume across Enterprise customers in the 30 days before migrating to IntelliSense against the 30 days after — controlling specifically for incidents that were genuinely user-impacting, so the comparison isn't inflated by simply alerting less often across the board.
The median reduction across that customer set was 94%. The lowest was 87%, the highest 98% — and critically, the real incident detection rate did not fall. IntelliSense caught the same significant incidents the previous threshold-based system caught. It just stopped alerting on the 94% of pages that never needed a human.
That result shows up downstream in customer-reported outcomes as well. One fintech on-call lead described the change directly: on-call pages dropped from roughly 21 a week to a handful within the first 30 days. A platform engineering team running dashboards across CloudWatch, Datadog, and Grafana — three tools lighting up for the same incident, with no single answer among them — reported MTTR falling from 3.2 hours to 18 minutes in the first month after consolidating onto one causal view.
None of these are edge cases. The pattern that repeats across customer feedback is the same one the arithmetic in Chapter 3 predicts: when the false-positive rate drops by an order of magnitude, the tax collected from the team drops by roughly the same order of magnitude, because the tax was never proportional to real incident volume — it was proportional to noise.
The direct engineering-hours cost of alert fatigue is the easiest to model, which is why Chapter 3 focused on it. The harder cost to put a number on is what chronic, low-signal on-call does to trust — both an engineer's trust in the tooling, and an organization's ability to retain people willing to carry a pager at all.
An on-call rotation that pages 21 times a week for mostly nothing doesn't just cost engineering hours; it trains people to treat the pager as background noise, which is the precise condition under which a real incident gets missed or delayed. Reducing false positives isn't only a productivity intervention. It's what makes on-call something an engineer can actually trust enough to act on quickly when it matters.
You don't need Applicare deployed to start measuring your own version of this. The method that produced the 94% figure in Chapter 5 is straightforward enough to run against your existing alerting data:
Whatever number that produces, it will very likely be more persuasive to your own leadership than any industry-wide statistic — because it's a number your own pager generated.