Model Distillation

Overview

Model distillation (knowledge distillation) is a training technique in which a smaller "student" model is trained to replicate the outputs of a larger "teacher" model rather than learning directly from raw data. The student cluster queries the teacher model for an effectively infinite number of questions, learning to give the same answers without learning all of the teacher's internal behaviors or requiring the full original training dataset. This dramatically reduces compute and data requirements for building a capable LLM: instead of acquiring a Google-scale data lake (pipeline stage 1) and training a large model from scratch (stage 2), a new entrant can fine-tune on proprietary data or use distillation to skip both stages entirely.

DeepSeek's 2025 open-source publication of its distillation methodology was a market-inflection event: by making the technique accessible to all organizations, DeepSeek democratized AI entry. Companies previously locked out by training infrastructure costs could now build capable LLMs using a modest inference cluster to query a frontier model. Supermicro's Johnson Jia used DeepSeek as the canonical example of this shift when advising enterprises on AI infrastructure strategy at OCP Global Summit 2025.