A Functional Taxonomy of World Models
Renderers, Simulators, Planners, and the Loop That Connects Them
America | Tech | Opinion | Culture | Charts
Today, we’re excited to be publishing an excerpt from Dr. Fei-Fei Li’s recent blogpost on World Models. You can read more on Fei-Fei’s Substack or on the World Labs website. Enjoy!
“The world is everything that is the case.”
— Ludwig Wittgenstein, Tractatus Logico-Philosophicus, 1921
The world is not made of words.
In an earlier essay, we argued that spatial intelligence is AI’s next frontier and that world models are the path to it. Here, the World Labs team and I want to go one level deeper: of the many things now being built and called ‘world models,’ which functional pieces actually compose that capacity — and what is each one for?
Language models have given machines an extraordinary command of concepts, vocabulary, and reasoning, but the physical world, virtual or real, runs on a different substrate. Where language models learn the statistical structure of text, world models learn the statistical structure of space and time: how light falls on a surface, how a garden looks from an angle no camera has captured, how objects respond to force and follow the laws of physics.
That makes “world model” one of the most important and most overloaded terms in AI today. Computer vision, robotics, reinforcement learning, and generative AI each claim to be building world models, and each means something quite different. A video model that produces gorgeous but physically impossible flames, a language model improvising a playable game, and a physics engine that faithfully simulates combustion all go by the same name.
The ancient Greeks could never agree on what the world was made of, whether fire, water, or indivisible atoms, because “world” was never a single thing. It was always a stand-in for whatever totality a given thinker needed to reason about. AI has inherited the same problem, at exactly the moment when the field needs precision.
The loop beneath the taxonomy
Cutting through that confusion starts with a diagram older than any of the technology in question. Reinforcement learning textbooks, including the canonical Sutton and Barto, have used a version of the same picture for decades to describe how an agent interacts with a world. The formal name for this picture is the partially observable Markov decision process, or POMDP, and the original definition of the term “world model” belongs to that tradition.
An agent, which can be a person, a robot, or a software system, takes actions. Those actions affect the state of the world. The agent never sees the state directly. What reaches the agent are observations: the photons that fall on a retina, the readings from a sensor, and the pixels in a video frame. New observations inform new actions, and the loop continues.
The word “state” needs unpacking, because the meaning shifts from field to field. This is not the chemist’s state, the difference between solid, liquid, and gas. This is the physicist’s and roboticist’s state: a complete description of what is happening in the world at a given moment, including every object, every position, every velocity, every property. State is the underlying reality of the world; complete in principle, but never directly visible to any agent inside it. Observations are an agent’s partial view of that reality. Actions are what the agent does in response.
This loop — agent to action to state to observation and back — is the structure that gave the modern term “world model” its technical meaning. The phrase itself is older, traced to Kenneth Craik’s 1943 proposal that minds reason by running “small-scale models” of reality, and carried into neural networks by the late 1980s and early 1990s. And the loop also explains what people mean by the term today. The different things now being called world models are in fact different projections of this same loop. Each one outputs a different piece of it.
This newsletter is provided for informational purposes only, and should not be relied upon as legal, business, investment, or tax advice. Furthermore, this content is not investment advice, nor is it intended for use by any investors or prospective investors in any a16z funds. This newsletter may link to other websites or contain other information obtained from third-party sources - a16z has not independently verified nor makes any representations about the current or enduring accuracy of such information. If this content includes third-party advertisements, a16z has not reviewed such advertisements and does not endorse any advertising content or related companies contained therein. Any investments or portfolio companies mentioned, referred to, or described are not representative of all investments in vehicles managed by a16z; visit https://a16z.com/investment-list/ for a full list of investments. Other important information can be found at a16z.com/disclosures. You’re receiving this newsletter since you opted in earlier; if you would like to opt out of future newsletters you may unsubscribe immediately.
After reading through the comments on both posts, a few themes seem to keep appearing: How does representation stay aligned with reality? How does a system know what it is allowed to do? Does self-modeling eventually imply consciousness? These are all downstream of coherence over time and even further downstream of the question: how does an intelligent system develop judgment and constraint in the first place?
My own research suggests that humans do not navigate the world through prediction alone. We develop constraints, judgments, and boundaries that help determine what information is trusted, when beliefs should be revised, and which actions remain acceptable as circumstances change.
Observationally, analogous mechanisms are the only starting point I've seen work in the wild, and remain the most promising path I've found toward a comprehensive solution.
Grounding the whole thing in the POMDP loop is what makes this click. A "world model" had turned into a Rorschach test where the vision, RL, and generative crowds each saw their own work in it. Treating renderer / simulator / planner as different projections of the same agent–state–observation loop turns a buzzword fight into an actual taxonomy.