After reading through the comments on both posts, a few themes seem to keep appearing: How does representation stay aligned with reality? How does a system know what it is allowed to do? Does self-modeling eventually imply consciousness? These are all downstream of coherence over time and even further downstream of the question: how does an intelligent system develop judgment and constraint in the first place?
My own research suggests that humans do not navigate the world through prediction alone. We develop constraints, judgments, and boundaries that help determine what information is trusted, when beliefs should be revised, and which actions remain acceptable as circumstances change.
Observationally, analogous mechanisms are the only starting point I've seen work in the wild, and remain the most promising path I've found toward a comprehensive solution.
I agree. I find that world models of today often still carry implicit biases and are too free about how their belief roams latent space. I’m looking forward to seeing future work that aim towards addressing this problem.
Here’s a related construct to consider when thinking about models: processes.
Models are simplified representations of something. They help explain, predict, organize, or guide thinking about a system.
Example: A model of well-being might include biology, emotions, thoughts, relationships, environment, and behavior.
Processes are sequences or patterns of activity through which something happens or changes over time.
Example: A person experiences a trigger, interprets it, feels emotion, chooses a coping response, and experiences an outcome.
Process models combine the two. They are structured representations of how a process unfolds. They show the key parts, relationships, stages, feedback loops, decision points, and possible outcomes.
Example: An RMW360-style process model might show how a situation activates an EMOT (emotional thought or trigger), which interacts with mindsets, emotions, coping strategies, behaviors, and resulting well-being.
Key question indeed : but do we have something else than models as meaningful abstraction for the human activity? Considering that since Heraclitus, the process was a visible fork but...
I'm not sure if this answers your question, but "process models" is what I propose.
For example, when we try to map the inner workings of human growth, cognition, or development, we face the ultimate process modeling challenge. We aren't just tracking a physical object moving through space; we are tracking the evolution of meaning inside a human mind.
On paper, DIKUW is often drawn as a pyramid; a stack of static layers. But when applied to human activity, it is actually a dynamic pipeline of transmutation. It models how raw, chaotic inputs are progressively refined into highly adaptive, value-driven behaviors over time.
The Dynamic Engine of DIKUW
If we treat DIKUW not as five separate filing cabinets, but as a continuous flow, we can see how a human processes change:
Data (The Flow of Sensation): This is the raw, uninterpreted flux of the universe hitting our senses or sensors—meaningless signals and noise. It is purely Heraclitean; a constant, chaotic stream of "becoming.
"Information (The First Freeze): We assign structure or context. We group the raw signals into patterns. The noise becomes "data in-formation.
Knowledge (The Functional Map): We combine information with experience to build actionable rules. This tells us how to react to the patterns we've identified.
Understanding (The Systemic Bridge): This is the crucial, often-missed transition engine. Understanding doesn't just look at rules; it grasps the why and the what-if. It maps the relationships between variables, allowing us to predict how the system will react when things change.
Wisdom (The Adaptive Outcome): Wisdom is the final integration. It is knowledge and understanding shaped by human values, ethics, and experience, resulting in adaptive, positive action in a constantly shifting world.
How DIKUW Solves the "Heraclitean Trap
"Returning to our original philosophical puzzle. How do we model a process without freezing it? The DIKUW framework offers a beautiful solution by focusing on transmutation rather than states.
1. It Models "Upward" Feedback Loops
In a static model, you move strictly from bottom to top. In a dynamic human process, the flow is circular. Once a person gains Wisdom or Understanding, that cognitive state acts as a filter that completely changes how they perceive and select raw Data at the very bottom. The Cognitive Loop: Your current level of wisdom dictates what noise you pay attention to, instantly reshaping the inputs entering your system.
2. It Highlights the "Transition States"
The magic of a process model isn't in the boxes (Data, Knowledge, etc.); it is in the arrows. How does a person move from Knowledge (knowing a rule) to Understanding (knowing why the rule exists)? How do they convert the distress of chaotic, conflicting Information into the adaptive, grounded clarity of Wisdom?
By focusing on the verbs (organizing, contextualizing, synthesizing, reflecting) rather than the nouns, cognitive models like DIKUW allow us to map human growth as an ongoing, living journey of adaptation.
I remember while I was doing my PhD back in 2015, to be the first one in the lab to make an end-to-end simulation in Gazebo, to be able to simulate our drones there. Even if it wouldn’t be exactly as reality, it could prevent us from making early mistakes in our algorithms.
A year later, I was using RL (DDPG algorithm in particular) to train an end-to-end controller for my drone to land on moving platforms.
I remember being super excited about simulations. I was totally sure that was the future. At IROS conference, I once pitched a generative model that would replace simulations, to a startup. They told me that was impossible. Light has to be algorithmically simulated, if not, it won’t behave realistically. They couldn’t be more wrong, seeing current visual world models.
Fast forward to 2026, I continue to see the trend. We have to get a mixture of physics simulations and visual world models. We need to create worlds with physics, and the renderer backbone has to be a neural network, given its proven capacity to absorb multimodal information.
I 100% agree with Fei Fei Li’s thesis, and I continue training robots with RL in simulation to this day (and I share many experiments here in Substack too).
This is an essential mapping of the landscape. By formally distinguishing between the Renderer, Simulator, and Planner, you’ve provided the taxonomy the field has been desperately lacking.
However, there is a structural risk in the trajectory toward a 'unified' model: the Renderer Illusion. If we collapse these functions without explicit epistemic boundary conditions, we risk 'stochastic laundering', where statistical plausibility is mistaken for physically anchored simulation. Without a hard boundary, the unified system doesn’t become a simulator; it becomes a closed-loop Spurious Stochastic Process optimizing inside a hallucinated environment.
The architectural "antidote kit" for this crisis, specifically the Non-Circularity Principle (NCP) and the Player-Frame Restriction (PFR) to show how we can achieve true spatial intelligence without falling into this feedback trap, is formalized here:
These aren't just theoretical concerns; they are the necessary architectural requirements for any world model that aims to be more than just a beautiful, but ungrounded, renderer.
We can never develop a near perfect world models with the "renderer" generating observations based on constraints it is optimized to which is fed into the physics sim that acts upon those observations to bring them close to reality and the planner making action-decisions from the reality it observes from those sim.
The world is a continuously changing process or ever evolving dynamics with lot of unpredictable uncertainties or stochasticity. We can never model the real world, but we can make these world models smart enough to understand this ever changing process through "stochasticity/uncertainty" that evolves from the interaction of these 3 components.
Going back to the cup example, renderer should generate the cup in every angle , and do you think there are finite angles to see that cup ? The physics tries to model the "cup falling" and do we know how many possible ways the cup could fall from a counter-top? The planner is going to "catch the cup" but will it know about the gust of wind that might change the trajectory of the fall ? The world is very uncertain and world models will be the never ending process. However, the first step could be to make the model know about the known physics of that fall and the countless other physics laws that will emerge as the cup falls from the table.
The taxonomy that matters for marketers: which world models treat your brand as signal vs. noise — GEO is really about earning a place in AI’s representation of reality.
The renderer/simulator/planner distinction is the most clarifying framing I've seen for this field — and I'd argue its most immediate impact isn't robotics, it's education.
Your point that pixels and state are different things is exactly what academia hasn't internalized. We still teach physics and engineering mostly through static observations: textbook figures, slides, pre-rendered animations. Those are all renderers. The student sees what the world looks like but never touches its state. Real learning happens in the loop you describe — act, perturb the state, observe the consequence, revise your internal model. Craik's "small-scale models of reality" are what education is supposed to build in a student's head, and you can't build them from observations alone.
As an example: my friend Farhad and I, using vibe coding tools, built an interactive walkthrough of space plasma physics and how it's simulated — https://sinaghazi.com/space-plasma-simulation. One section lets learners push a numerical solver past its stability limit and watch a solution that still looks plausible while its underlying state is already garbage: your renderer-vs-simulator argument made visceral. A few years ago this would have taken a funded team and a semester. Two people built it in spare time.
That's the part academia should sit with. The cost of building explorable, physically grounded learning worlds is collapsing — first through AI-assisted coding, and soon through the world models you describe. The institutions that treat this as core pedagogy rather than a side demo will define the next era of science education.
The simulator-to-planner loop in this taxonomy gets stress-tested fastest through deployment volume, not lab scale. China's current physical AI buildout is the largest real-world validation set for exactly this: Unitree has roughly 50,000 robots active in Chinese industrial settings, CATL's manufacturing AI runs on domestic semiconductor and sensor stacks, and BYD is standing up a 50,000-dealer humanoid deployment network. The question of how a world model keeps its representation aligned with physical reality gets answered under conditions those deployments generate. Chinese labs working on physical AI (Unitree, Galbot, Agibot) are sitting on the feedback data that academic simulators approximate. The loop Li describes — renderer, simulator, planner — is being iterated fastest in environments Western frontier labs mostly don't have direct access to. We track this at China AI Dispatch.
Grounding the whole thing in the POMDP loop is what makes this click. A "world model" had turned into a Rorschach test where the vision, RL, and generative crowds each saw their own work in it. Treating renderer / simulator / planner as different projections of the same agent–state–observation loop turns a buzzword fight into an actual taxonomy.
Grounding the whole thing in the POMDP loop is what makes this click: "world model" had turned into a Rorschach test where the vision, RL, and generative crowds each saw their own work in it. Treating renderer / simulator / planner as different projections of the same agent–state–observation loop turns a buzzword fight into an actual taxonomy.
After reading through the comments on both posts, a few themes seem to keep appearing: How does representation stay aligned with reality? How does a system know what it is allowed to do? Does self-modeling eventually imply consciousness? These are all downstream of coherence over time and even further downstream of the question: how does an intelligent system develop judgment and constraint in the first place?
My own research suggests that humans do not navigate the world through prediction alone. We develop constraints, judgments, and boundaries that help determine what information is trusted, when beliefs should be revised, and which actions remain acceptable as circumstances change.
Observationally, analogous mechanisms are the only starting point I've seen work in the wild, and remain the most promising path I've found toward a comprehensive solution.
I agree. I find that world models of today often still carry implicit biases and are too free about how their belief roams latent space. I’m looking forward to seeing future work that aim towards addressing this problem.
Here’s a related construct to consider when thinking about models: processes.
Models are simplified representations of something. They help explain, predict, organize, or guide thinking about a system.
Example: A model of well-being might include biology, emotions, thoughts, relationships, environment, and behavior.
Processes are sequences or patterns of activity through which something happens or changes over time.
Example: A person experiences a trigger, interprets it, feels emotion, chooses a coping response, and experiences an outcome.
Process models combine the two. They are structured representations of how a process unfolds. They show the key parts, relationships, stages, feedback loops, decision points, and possible outcomes.
Example: An RMW360-style process model might show how a situation activates an EMOT (emotional thought or trigger), which interacts with mindsets, emotions, coping strategies, behaviors, and resulting well-being.
Key question indeed : but do we have something else than models as meaningful abstraction for the human activity? Considering that since Heraclitus, the process was a visible fork but...
I'm not sure if this answers your question, but "process models" is what I propose.
For example, when we try to map the inner workings of human growth, cognition, or development, we face the ultimate process modeling challenge. We aren't just tracking a physical object moving through space; we are tracking the evolution of meaning inside a human mind.
The classic way to model this evolution is through cognitive hierarchies, most notably, the DIKUW (Data, Information, Knowledge, Understanding, Wisdom) framework. https://stephenbellerphd.substack.com/p/from-data-overload-to-wisdom
On paper, DIKUW is often drawn as a pyramid; a stack of static layers. But when applied to human activity, it is actually a dynamic pipeline of transmutation. It models how raw, chaotic inputs are progressively refined into highly adaptive, value-driven behaviors over time.
The Dynamic Engine of DIKUW
If we treat DIKUW not as five separate filing cabinets, but as a continuous flow, we can see how a human processes change:
Data (The Flow of Sensation): This is the raw, uninterpreted flux of the universe hitting our senses or sensors—meaningless signals and noise. It is purely Heraclitean; a constant, chaotic stream of "becoming.
"Information (The First Freeze): We assign structure or context. We group the raw signals into patterns. The noise becomes "data in-formation.
Knowledge (The Functional Map): We combine information with experience to build actionable rules. This tells us how to react to the patterns we've identified.
Understanding (The Systemic Bridge): This is the crucial, often-missed transition engine. Understanding doesn't just look at rules; it grasps the why and the what-if. It maps the relationships between variables, allowing us to predict how the system will react when things change.
Wisdom (The Adaptive Outcome): Wisdom is the final integration. It is knowledge and understanding shaped by human values, ethics, and experience, resulting in adaptive, positive action in a constantly shifting world.
How DIKUW Solves the "Heraclitean Trap
"Returning to our original philosophical puzzle. How do we model a process without freezing it? The DIKUW framework offers a beautiful solution by focusing on transmutation rather than states.
1. It Models "Upward" Feedback Loops
In a static model, you move strictly from bottom to top. In a dynamic human process, the flow is circular. Once a person gains Wisdom or Understanding, that cognitive state acts as a filter that completely changes how they perceive and select raw Data at the very bottom. The Cognitive Loop: Your current level of wisdom dictates what noise you pay attention to, instantly reshaping the inputs entering your system.
2. It Highlights the "Transition States"
The magic of a process model isn't in the boxes (Data, Knowledge, etc.); it is in the arrows. How does a person move from Knowledge (knowing a rule) to Understanding (knowing why the rule exists)? How do they convert the distress of chaotic, conflicting Information into the adaptive, grounded clarity of Wisdom?
By focusing on the verbs (organizing, contextualizing, synthesizing, reflecting) rather than the nouns, cognitive models like DIKUW allow us to map human growth as an ongoing, living journey of adaptation.
I remember while I was doing my PhD back in 2015, to be the first one in the lab to make an end-to-end simulation in Gazebo, to be able to simulate our drones there. Even if it wouldn’t be exactly as reality, it could prevent us from making early mistakes in our algorithms.
A year later, I was using RL (DDPG algorithm in particular) to train an end-to-end controller for my drone to land on moving platforms.
I remember being super excited about simulations. I was totally sure that was the future. At IROS conference, I once pitched a generative model that would replace simulations, to a startup. They told me that was impossible. Light has to be algorithmically simulated, if not, it won’t behave realistically. They couldn’t be more wrong, seeing current visual world models.
Fast forward to 2026, I continue to see the trend. We have to get a mixture of physics simulations and visual world models. We need to create worlds with physics, and the renderer backbone has to be a neural network, given its proven capacity to absorb multimodal information.
I 100% agree with Fei Fei Li’s thesis, and I continue training robots with RL in simulation to this day (and I share many experiments here in Substack too).
I admire all your work! Wish you all the best!
This is an essential mapping of the landscape. By formally distinguishing between the Renderer, Simulator, and Planner, you’ve provided the taxonomy the field has been desperately lacking.
However, there is a structural risk in the trajectory toward a 'unified' model: the Renderer Illusion. If we collapse these functions without explicit epistemic boundary conditions, we risk 'stochastic laundering', where statistical plausibility is mistaken for physically anchored simulation. Without a hard boundary, the unified system doesn’t become a simulator; it becomes a closed-loop Spurious Stochastic Process optimizing inside a hallucinated environment.
The architectural "antidote kit" for this crisis, specifically the Non-Circularity Principle (NCP) and the Player-Frame Restriction (PFR) to show how we can achieve true spatial intelligence without falling into this feedback trap, is formalized here:
https://trissimondsen.wordpress.com/2026/07/19/the-boundary-conditions-of-unified-world-models-why-simulation-fails-without-player-frame-restrictions/
These aren't just theoretical concerns; they are the necessary architectural requirements for any world model that aims to be more than just a beautiful, but ungrounded, renderer.
Exploration of the concept of Planner-Simulator-Renderer as an alternative to JEPA’s Model Predictive Control.
We can never develop a near perfect world models with the "renderer" generating observations based on constraints it is optimized to which is fed into the physics sim that acts upon those observations to bring them close to reality and the planner making action-decisions from the reality it observes from those sim.
The world is a continuously changing process or ever evolving dynamics with lot of unpredictable uncertainties or stochasticity. We can never model the real world, but we can make these world models smart enough to understand this ever changing process through "stochasticity/uncertainty" that evolves from the interaction of these 3 components.
Going back to the cup example, renderer should generate the cup in every angle , and do you think there are finite angles to see that cup ? The physics tries to model the "cup falling" and do we know how many possible ways the cup could fall from a counter-top? The planner is going to "catch the cup" but will it know about the gust of wind that might change the trajectory of the fall ? The world is very uncertain and world models will be the never ending process. However, the first step could be to make the model know about the known physics of that fall and the countless other physics laws that will emerge as the cup falls from the table.
Code World Models combine the best of handcrafted heuristics with learned predictive models that capture the underlying dynamics of the environment.
https://jdsemrau.substack.com/p/deepmind-code-world-models
Amazing. Any of your students are founders. Welcome to Founders Castle https://discord.gg/qqs9ENFhG
The taxonomy that matters for marketers: which world models treat your brand as signal vs. noise — GEO is really about earning a place in AI’s representation of reality.
Deepest level of understanding is always the most economical. To know more - https://anirudhchakrahari.substack.com/p/the-deepest-knowing-might-be-the?utm_source=share&utm_medium=android&r=28cxbt
The renderer/simulator/planner distinction is the most clarifying framing I've seen for this field — and I'd argue its most immediate impact isn't robotics, it's education.
Your point that pixels and state are different things is exactly what academia hasn't internalized. We still teach physics and engineering mostly through static observations: textbook figures, slides, pre-rendered animations. Those are all renderers. The student sees what the world looks like but never touches its state. Real learning happens in the loop you describe — act, perturb the state, observe the consequence, revise your internal model. Craik's "small-scale models of reality" are what education is supposed to build in a student's head, and you can't build them from observations alone.
As an example: my friend Farhad and I, using vibe coding tools, built an interactive walkthrough of space plasma physics and how it's simulated — https://sinaghazi.com/space-plasma-simulation. One section lets learners push a numerical solver past its stability limit and watch a solution that still looks plausible while its underlying state is already garbage: your renderer-vs-simulator argument made visceral. A few years ago this would have taken a funded team and a semester. Two people built it in spare time.
That's the part academia should sit with. The cost of building explorable, physically grounded learning worlds is collapsing — first through AI-assisted coding, and soon through the world models you describe. The institutions that treat this as core pedagogy rather than a side demo will define the next era of science education.
The simulator-to-planner loop in this taxonomy gets stress-tested fastest through deployment volume, not lab scale. China's current physical AI buildout is the largest real-world validation set for exactly this: Unitree has roughly 50,000 robots active in Chinese industrial settings, CATL's manufacturing AI runs on domestic semiconductor and sensor stacks, and BYD is standing up a 50,000-dealer humanoid deployment network. The question of how a world model keeps its representation aligned with physical reality gets answered under conditions those deployments generate. Chinese labs working on physical AI (Unitree, Galbot, Agibot) are sitting on the feedback data that academic simulators approximate. The loop Li describes — renderer, simulator, planner — is being iterated fastest in environments Western frontier labs mostly don't have direct access to. We track this at China AI Dispatch.
Grounding the whole thing in the POMDP loop is what makes this click. A "world model" had turned into a Rorschach test where the vision, RL, and generative crowds each saw their own work in it. Treating renderer / simulator / planner as different projections of the same agent–state–observation loop turns a buzzword fight into an actual taxonomy.
Grounding the whole thing in the POMDP loop is what makes this click: "world model" had turned into a Rorschach test where the vision, RL, and generative crowds each saw their own work in it. Treating renderer / simulator / planner as different projections of the same agent–state–observation loop turns a buzzword fight into an actual taxonomy.
thanks for the post. which of the three layers do you think is most under-funded right now?
Excellent post. Thank you