OCP NVMe Latency Monitoring

Overview

OCP NVMe Latency Monitoring is an OCP-specified SSD feature that provides drive-side capture of tail-latency events and debug logs, closing the gap left by host-level kernel monitoring that missed extreme tail latencies (P99.999999+) at hyperscale, where events were transient, intermittent, hard to reproduce, and typically had no debug logs available by the time a problematic drive was identified.

The feature sorts each I/O operation (read, write, TRIM) into one of four configurable latency buckets with host-defined thresholds; Meta uses 100ms, 250ms, 500ms, and 1000ms as its four bucket boundaries and focuses on events over 1 second as its primary concern. It exposes three active counters (Active Bucket Counter, Active Measured Latency, Active Latency Time Stamp) for current-window tracking and three static counters (Static Bucket Counter, Static Measured Latency, Static Latency Time Stamp) that persist across resets and power cycles, enabling long-term trend analysis and retroactive correlation of stall events with host or application logs. A CLI latency monitoring log parser is available in nvme-cli on GitHub for parsing the C3 Log Page data and identifying drives that exhibited stall events.

In an 8-day fleet sample, Meta detected 6,700 distinct drives with stall events (I/O over 1 second); a single unidentified vendor (Vendor C) accounted for 6,716 such events (~90% of the total), while Vendors A, B, and D showed only 3, 14, and 7 respectively.