Flash in Fleet

Meta engineers Vineet Parekh and Minh Hoang presented two complementary advances in Meta's fleet-scale NVMe SSD quality program at OCP Global Summit 2025. The first segment shared Meta's production deployment of OCP NVMe Latency Monitoring across thousands of SSDs, including the first public fleet-wide stall data. The second segment covered Meta's complete open-sourcing of SSD qualification test cases via the OCP Diag AutoVal framework on GitHub, which has cut firmware regression cycles from 8 to 6 weeks.

Key Takeaways

• Meta has enabled OCP NVMe Latency Monitoring on thousands of production SSDs; in an 8-day sample, 6,700 distinct drives exhibited stall events (I/O over 1 second), with one unidentified vendor contributing approximately 6,716 of those events (~90%).
• OCP Latency Monitoring uses four configurable latency buckets (100ms/250ms/500ms/1000ms), dual active/static counter sets, and persistent storage across resets—enabling precise timestamp correlation and retroactive vendor debug log sharing.
• Meta open-sourced all SSD qualification test cases via OCP Diag AutoVal on GitHub by June 2025, cutting firmware regression from 8 to 6 weeks (25% reduction) and achieving 100% test consistency across Meta, ODMs, and SSD vendors.
• The AutoVal rollout timeline ran from Hi5 discontinuation (November 2023) to NDA test sharing (Q1 2024), public AutoVal release (May 2024), full vendor alignment (November 2024), and all tests on GitHub (June 2025); SSD security test cases are planned for Q4 2025.
• Running the same shared test cases across Meta, ODM, and SSD vendor labs ('left shifting') allows vendors to catch and fix defects before Meta allocates regression resources, reducing firmware churn from multi-platform issues.