Time to First Token

Overview

Time to first token (TTFT) is the elapsed time from when a user or application submits an inference request to a language model to when the model returns the first token of its response. For interactive AI applications — chatbots, code assistants, real-time summarization — TTFT is the primary user-perceived latency metric and a key commercial differentiator. As AI inference has scaled from experimental deployments to production systems serving millions of concurrent users, TTFT has become the central performance objective driving data center architecture decisions.

TTFT pressure is reshaping infrastructure design: it motivates disaggregated inference architectures (separating the compute-intensive prefill phase from the memory-bandwidth-intensive decode phase onto different rack types), drives demand for higher-bandwidth interconnects between racks, and creates urgency around deployment speed. A cluster that takes 12-18 months to deploy represents approximately half the GPU card's lifetime value foregone, making time-to-first-token a financial metric as much as a latency metric.