Alert Fatigue in Hyperscale Environments: A Metrics-Based Approach to Signal Tuning in Open Networking

Author: Puja Gupta (presenter) Collection: 2025-OCP-Global-Summit-Networking

Summary

Alert fatigue in hyperscale networks occurs when a single physical failure — such as one ToR port going down — cascades into thousands of correlated alerts across server, ToR, fabric, spine, and resource hub layers simultaneously. At hyperscale, a single cloud operates millions of physical network devices, meaning that a single contained failure can appear as a system-wide crisis to on-call engineers. The presenter introduces a metrics-based framework for signal tuning centered on three key operational metrics: Time to Detect (TTD), Time to Mitigate (TTM), and Time to Notify (TTN). Improving each metric requires both smarter alerting aggregation and AI-driven automation.

The proposed architecture flows telemetry from OCP hardware (ToR, Spine, Fabric switches emitting port counters, error rates, and link state) through a correlation engine (statistical/ML-driven) to AI/agentic solutions. Noise reduction is achieved through two aggregation strategies: device-level aggregation (collapsing 30–80 link-level alerts into one device incident over a rolling 2–5 minute window) and span-level aggregation for longer-duration events. The presenter demonstrates an experimental agentic workflow for automated troubleshooting: event trigger → planning → tool-use (run commands on devices) → reflection → result assessment, with 90% improvement in operational efficiency observed in early experiments. Open-source tools cited include Ansible/AWX, StackStorm, and LangChain for predefined automated remediation, and OpenTelemetry with Grafana/Tempo/Loki plus Elasticsearch + LLM agents for ticket enrichment.