AI Inference Memory Hierarchy

Overview

The AI inference memory hierarchy describes the layered arrangement of HBM (High Bandwidth Memory), DRAM, and NAND storage that must be co-optimized to maximize tokens per second per megawatt (TPS/MW). As of 2025, reasoning-capable models have driven token output roughly 10x higher — Grok 4 generates 110M output tokens versus Claude 4 Opus at 11M — shifting the primary performance constraint from compute FLOPs to memory bandwidth and token throughput.

SK hynix articulates a two-step optimization roadmap: first, smarter integration of next-generation DRAM and SSD with HBM3/3E to improve token throughput per QoS target; second, deploying AI-optimized memory and storage alongside HBM4/4E to push beyond current limits. The AI data pipeline segments memory and storage requirements by stage: data ingestion and archive are capacity-intensive (QLC SSD, High Capacity Storage); training is compute and memory intensive (HBM, CXL-MEM); inference requires both high compute and high memory/storage bandwidth (TLC SSD, HBM). Each stage maps to distinct product requirements across HBM, CXL, and NAND tiers.