Verifiability and Automation

Overview

Verifiability and Automation is a predictive framework articulated by Andrej Karpathy for understanding which domains LLMs will automate fastest. The core thesis: traditional computers automate what you can specify in code; LLMs automate what you can verify. Domains with automatic verification mechanisms — test suites, mathematical proofs, game outcomes — receive the most reinforcement learning investment and therefore see the fastest capability gains.

This framework explains why code and math were the first domains where LLMs achieved superhuman-adjacent performance: they have natural verification environments (compilers, test suites, formal proofs). It also predicts that any domain where verification can be constructed — even imperfectly, such as councils of LLM judges for writing quality — can eventually be automated. The framework is closely related to the concept of Jagged Intelligence and the broader transition toward Agentic Engineering, where verification becomes the critical bottleneck for automated software development loops.