Efficiently Powering The Next Wave of Agentic Reasoning Applications: Making Gen AI Attainable

Author: Keith Nellis, Director of Marketing, d-Matrix Collection: 2025 OCP Global Summit - Expo Hall Sessions

Summary

d-Matrix presents its Digital In-Memory Compute (DIMC) architecture as a purpose-built solution for AI inference, arguing that the GPU-dominated training era is giving way to an inference era where latency, opex, and multimodality matter more than raw capex-driven compute. The company's core thesis is that the "memory wall" -- where compute capability growth outpaces memory bandwidth -- is the fundamental bottleneck for AI inference, and that DIMC breaks through this by combining memory and compute on a single silicon substrate, achieving approximately 150 TB/s memory bandwidth versus 8-16 TB/s for state-of-the-art GPU HBM solutions.

The company's first product, Corsair, is shipping now as a 600W air-cooled PCIe Gen5 card built on TSMC 6nm with 8 chiplets. It delivers 2,400 TFLOPS at MXINT8 (9,600 TFLOPS at MXINT4), includes 2GB SRAM at 150 TB/s and 256GB LPDDR5 at 400 GB/s, with PCIe at 128 GB/s and DMX Bridge at 512 GB/s. Performance benchmarks show 1ms time-per-output-token (TPOT) on Llama3 8B at 45,000 tokens/s throughput, and 2ms TPOT on Llama3 70B at 30,000 tokens/s -- claimed to be 10x faster than GPU alternatives.

The roadmap extends from Corsair (DIMC) to Raptor (3DIMC Single Stack with 3D DRAM logic stacking, announced at Hot Chips) to Lightning (3DIMC Multi-Stacked). The IO roadmap progresses from JetStream (current Ethernet NIC with 2-microsecond latency, 400 Gbps, 150W TDP) to ESUN/UALink Electrical IO Chiplet and ultimately ESUN/UALink Optical IO Chiplet. SquadRack, a rack-scale reference design announced at this show with partners Super Micro, Broadcom, and Arista, enables both inference-as-a-service and large proprietary model deployment architectures.