Conceptualizing Next-Generation Memory & Storage Optimized For AI Inference

Author: Thomas Choy, Team Lead, Next Generation Memory and Storage Product Planning, SK hynix Collection: 2025 OCP Global Summit - Expo Hall Sessions

Summary

SK hynix presents a conceptual framework for how AI inference is fundamentally changing memory and storage requirements in two key ways. First, the need for energy-efficient offloading: LLM weights require reading hundreds of gigabytes with near-zero writes (O(M*N) linear but at massive model sizes), while KV caches grow quadratically with read-to-write ratios of 1,000:1 to 10,000:1 -- far different from general-purpose workloads. This drives interest in Processing-In-Memory (PIM) and Processing-Near-Memory (PNM) technologies. Second, access patterns are becoming "semi-sequential" for memory (not truly random due to predictable KV cache and weight access) and "semi-random" for storage (not truly sequential, with 512-byte granularity for GPU-initiated GenAI workloads).

SK hynix distinguishes PIM (bank-level parallelism with ALUs attached to each memory bank, offering up to 48x internal bandwidth versus external interface) from PNM (compute capability not tied to bank-level parallelism, can be a chiplet, stacked die, or module-level). Their GDDR6-based AIM (Accelerator-In-Memory) card targets on-device AI first (where NPU power budgets are most constrained) before data center applications. The CMMaX product line implements PNM using CXL interface, with an add-in card containing up to 512 GB DDR4, ARM Cortex A72 cores, and hardware engines for memory capacity expansion, demonstrating benefits in vector DB workloads using Facebook AI Similarity Search (FAISS). The presentation references Microsoft's managed retention memory research proposing optimization of read bandwidth and energy with relaxed retention requirements for terabyte-class storage-class memory.