Deterministic ISSU Qualification

Author: Ryan (presenter, organization not stated) Collection: 2025-OCP-Global-Summit-Networking

Summary

ISSU (In-Service Software Upgrade) — also called warm upgrade or warm reboot in SONiC — allows network switches to upgrade their software without dropping a single packet. ISSU qualification is the process of evaluating whether an ISSU upgrade was truly successful against a defined set of criteria and metrics. This talk defines the key concept of "data consistency" as the core quality indicator: after a warm upgrade from version A to version B, the resulting system state should be identical to a cold boot of version B. Inconsistencies in ASIC DB, State DB, Config DB, or hardware register state are called "differences," and differences compound across multiple sequential warm upgrades, representing increasing risk of data plane impact.

The presenter describes a two-group check methodology: (1) hardware vs. software checks — SAI GET calls to compare ASIC DB expected state vs. actual hardware register state, and (2) software warm boot vs. software cold boot — snapshot comparison of App DB, Config DB, and State DB. Current implementation covers only 5 SAI object types (pools, profiles, switches, WED, port-except-speed), yielding about 2,400 matching values out of 16,000 total (15%). Production data shows 50% of data-plane-impacting ISSU events come from SW/SW inconsistencies, 30% from SW/HW inconsistencies, with LACP Failure and ARP/ND Failure accounting for 10% each. The target state is 100% data consistency so that SONiC's extensive cold-boot test suite becomes directly applicable to warm-boot validation, dramatically increasing coverage confidence.