I've watched operators silence alert dashboards during site visits. Not because they're being careless — because the system has trained them to. When a pressure monitoring platform generates 150–300 threshold alerts per month and 95% of them resolve without action, the operators who respond to every alert aren't safer than the ones who've learned to filter; they're just more exhausted. The problem is a design failure in how the alert thresholds were configured, not a personnel failure.
Alert fatigue is the most common operational problem we encounter during the calibration phase of a new deployment. This post is a practical guide to diagnosing and fixing it — covering threshold configuration, confirmation windows, cross-parameter correlation rules, and scheduled suppression. These techniques apply to any distribution system monitoring platform, not just Watsynq.
Why default thresholds cause alert fatigue
Most monitoring platform deployments start with default alert thresholds that are technically correct but operationally useless. A typical example: a low-pressure alert configured at 20 PSI below the zone's normal operating pressure. That's a reasonable alarm point for a break scenario — but if the zone has a pump station that creates a 15-PSI swing during its normal start cycle, the alarm fires three times a day from normal operations. The operators learn within two weeks to ignore low-pressure alarms in that zone. When the real break happens at 3 a.m., the alarm fires and nobody acts on it for 40 minutes.
Default thresholds are also static. Water distribution systems have strong diurnal demand patterns — pressure is lower at peak hour than at 2 a.m., flow rates change dramatically, and transient events from fire suppression demand are episodic. A static threshold that's tight enough to catch a real event at peak hour will generate noise during off-peak hours, and vice versa.
The baseline-relative threshold approach
The most effective threshold model for distribution system monitoring is not an absolute setpoint — it's a deviation from a rolling statistical baseline. Instead of "alert when pressure falls below 40 PSI," configure the rule as "alert when pressure deviates below the 95th-percentile lower bound of the trailing 30-day distribution for this sensor at this time of day." This approach automatically adapts to the normal operational behavior of each measurement point and generates alerts only when something genuinely anomalous is occurring.
Building this baseline requires having the historical data to compute it, which is why configuring alert thresholds is properly a 30-day activity after data ingestion — not something you set on day one of deployment. During the first 30 days, you're observing the system's normal behavioral envelope before defining the alert boundary.
The baseline-relative approach works well for pressure and flow anomalies. For water quality parameters — chlorine residual in particular — absolute thresholds remain appropriate because the regulatory floor (EPA MRDL of 4.0 mg/L for chlorine, with distribution system residual requirements varying by state) is a meaningful minimum regardless of the sensor's historical baseline. You should not configure a chlorine alert that adapts upward if the chlorine residual has been chronically low in a particular zone; you should fix the chlorine residual.
Confirmation windows and alert suppression
A single threshold exceedance, sustained for 30 seconds, should not generate an alert with the same urgency as a sustained deviation lasting 10 minutes. Most real failure events produce signals that persist or intensify; most false alarms from pump cycling, valve actuation, or demand surges resolve within 2–5 minutes. Configuring a confirmation window — requiring the threshold exceedance to persist for a defined minimum duration before the alert fires — eliminates a large fraction of false alarms without meaningfully delaying detection of real events.
For pressure anomaly detection in a distribution main context, we typically recommend confirmation windows of 3–8 minutes for low-priority alerts and no confirmation window (immediate) for alerts that cross both a pressure threshold and a companion flow anomaly flag simultaneously. The two-parameter co-occurrence is a much stronger signal than either parameter alone.
Scheduled suppression addresses a different source of noise: known operational events that reliably generate anomalous readings. A quarterly fire hydrant flushing program in a residential district creates pressure drops that will trigger low-pressure alerts in the adjacent zone — but those drops are planned, known, and not indicative of a failure. Configure suppression windows for scheduled flushing programs, planned pump maintenance shutdowns, and any other regular operational activities that create expected pressure excursions. The SCADA operations schedule is the source for these — they should already be logged.
Cross-parameter correlation rules: separating signal from noise
Single-parameter thresholds are inherently noisy because any one measurement has multiple causal explanations. A pressure drop at a zone boundary measurement point could indicate: a main break in the zone, a fire department hydrant connection, a pump start transient in an adjacent zone, sensor drift, or a demand surge from an irrigation system activating. Most of these are not actionable emergencies.
Cross-parameter correlation rules increase specificity by requiring multiple signals to co-occur before triggering a high-priority alert. Effective combinations for break detection in distribution mains:
- Low pressure alert + sustained flow increase at the zone DMA boundary meter (indicates water loss, not just demand variation)
- Low pressure alert + elevated turbidity at the downstream WQ sensor (consistent with sediment disturbance from a break event)
- Pressure transient spike + risk score above 70 for the nearest high-risk segment (combines operational signal with model-based context)
When these co-occurrence rules are the trigger for the highest-priority alert tier, operators receive genuinely high-confidence alerts that warrant immediate response — and the lower-priority single-parameter alerts become informational, reviewed during business hours rather than generating overnight callouts.
Calibrating alert volume: a practical target
There's no universal correct answer for alert volume, but a reasonable operational target for a distribution system with 10–20 pressure monitoring points is 5–15 high-priority alerts per month that warrant immediate review, with no more than 2–3 actually requiring field response. If your high-priority alert rate is significantly above 15 per month, you have a threshold configuration problem. If it's below 3 per month and you're still having undetected break events, you have a coverage or sensitivity problem.
We track the alert-to-action conversion rate for each utility in the pilot cohort and flag deployments where that ratio is drifting outside the expected range. A sudden increase in alert volume without a corresponding increase in detected events is usually a signal that a sensor is drifting or a threshold needs tightening. A decrease in alert volume without network explanation is worth investigating before the next quarterly report — not after.
The operational trust problem
Alert systems that generate noise erode trust over time in a way that's hard to rebuild quickly. An operator who has learned over six months that a particular alert is 90% likely to be a false alarm will have a delayed response even after the threshold is corrected — the behavioral adaptation is stickier than the technical fix. Getting the alert configuration right early, and keeping it calibrated, is not just a technical preference — it's an operational safety issue.
When we do the 30-day calibration review with a new utility deployment, we specifically look at the operator response pattern: how quickly are alerts being acknowledged, and which alert types have the longest acknowledgment latency? Latency is a direct measure of operator confidence in the alert. Long latency = low confidence = threshold recalibration needed.
Marcus Tran is Customer Success Lead at Watsynq. He manages alert threshold calibration for all new utility deployments.