QUIC Kernel and Device Offload

Author: Eric Davis, System Architect, Broadcom; Eric Spada, Distinguished Engineer, Broadcom Collection: 2023 Global Summit - Networking

Summary

Broadcom engineers presented a detailed architecture for offloading QUIC protocol cryptographic operations to NIC hardware, building on the successful KTLS offload model. QUIC, now widely deployed for mobile applications (Facebook, Google Chrome) and data center workloads, runs as a user-space library over UDP and handles packetization, congestion control, reliability, and scheduling. Transport security encryption currently consumes 10-30% of CPU cycles in large deployments, and NIC inline offload can eliminate this overhead by leveraging the natural memory copy that occurs during RDMA or L2 operations.

The proposed architecture offloads only 1-RTT short header packet encryption/decryption to hardware, while long header packets (used for TLS 1.3 handshake and key exchange) remain in the application library. The design introduces a QUIC kernel module that maintains a hash table of offloaded flow contexts, uses netlink (rather than socket options) for communication between application and kernel, and assigns each offloaded flow a context ID (XID) that serves as a fast lookup key throughout the entire TX/RX path. This achieves only one lookup per direction, improving on KTLS which requires two lookups.

A key implementation challenge is QUIC's variable-length connection IDs (0-20 bytes), which are not self-describing in short headers. Hardware vendors must advertise which connection ID lengths they support. The design also addresses UDP GSO integration for high performance, requiring all packets in a large segment to target the same destination connection ID. Broadcom planned to upstream the device offload in their bnxt driver with QUIC kernel updates staged for Q1 2024, with Meta providing significant feedback on the design.