KV Cache

Overview

The KV (Key-Value) cache stores intermediate attention activations generated during LLM inference. In the decode phase, the GPU must repeatedly read both model weights and KV cache data to generate each successive token — making it a central memory bandwidth and capacity bottleneck. Token volume is projected to reach 300x current levels as reasoning models scale, compounding this pressure. In multi-turn inference, the KV cache grows with each conversation turn and must be reloaded from context start at each prefill stage, causing Time to First Token (TTFT) to degrade cumulatively as turns accumulate.

Storage access characterization: KV cache access follows a 4GB sequential-read (SR) pattern at medium throughput, contrasted with vector database search which uses 512-byte random-read (RR) at high throughput. SK hynix's AIN B (High Bandwidth Flash) is designed to hold the KV cache without spilling to CPU memory or conventional SSD, improving TTFT across multi-turn interactions.