Why AI needs world models

In July, World Labs reported a robot learning a two-arm box-packing task in simulation, then performing it on real hardware.

Someone still had to capture the real setup and check the simulation. But the robot's practice happened inside it.

For a physical machine, practice takes actual time. If something falls, someone picks it up. If the box gets damaged, someone replaces it. Even a successful attempt leaves somebody with a box to unpack.

Now imagine keeping the attempt that went wrong.

A flap caught. The robot missed the opening. Instead of waiting for that combination to happen again, you recreate it and let the machine practice around it.

That is the possibility I find most interesting in world models: difficult experience could become something we deliberately produce, instead of something we have to wait for.

I've been thinking about why this has people so excited. I suspect our most useful AI systems will combine world models and LLMs, with other systems supplying the tools and controls they need to act.

What the model is learning

A world model learns patterns in an environment and uses them to predict things it hasn't observed. The "world" might be no bigger than a packing table.

Suppose the robot sees a closed box. The image tells it about the shape and position. It doesn't necessarily tell it whether the box is empty. Two boxes can look identical and behave differently when pushed. Watching a push could supply evidence that another photograph cannot.

For this kind of task, the system learns from observations, actions and their results. It predicts what will happen, compares that with what happened, and adjusts. With enough useful examples, it can learn to anticipate the effect of a new action.

It needn't store a little animated warehouse inside itself. Numerical features can preserve information about position and movement without reproducing every mark printed on the cardboard.

An agent could use those predictions to compare possible movements before choosing one. It could also learn which actions work by practicing inside the model.

The term also covers a different kind of prediction.

World Labs' Atlas, announced on September 1, takes images and camera positions and generates views from new angles. It fills gaps in what was captured with what it has learned about scenes. Some of the results are imagined.

For a filmmaker, exploring the same set from different positions is already useful.

But moving a virtual camera around our box doesn't establish how much force it takes to move the box itself. Predicting views and predicting contact are different capabilities.

A spectacular demonstration can be a real achievement without proving both.

What language leaves open

Yann LeCun has been making a version of this argument for years.

In his 2024 interview with Lex Fridman, he argued that language alone leaves out too much of the physical knowledge an intelligent system needs. His proposed direction includes learning from observation and interaction, then predicting the consequences of actions.

A packing manual might say "hold the box firmly." For a person, that instruction leans on years of handling objects. The words don't specify every adjustment needed as the weight shifts.

A machine can give an excellent explanation of packing while still being unable to guide the gripper.

(That's why LLMs can write decent poetry, but can't help with your laundry.)

I wouldn't take the next step and say language models can only shuffle words, though.

Researchers trained a GPT-style model on Othello moves without supplying the rules. They found an internal representation of the board. When they changed that representation, the model's predictions changed too.

So "it predicts the next thing" doesn't settle what it has learned along the way. Predicting a sequence can involve learning something about the world that produced it.

That doesn't establish physical understanding. An Othello board won't tell our robot how hard to grip a box.

An LLM could interpret the packing instruction. A world model could help anticipate the motion. A controller could turn that into actions.

That's why I find the combination more interesting than another argument about which technology will kill the other.

A place to get it wrong

Engineers have used simulators for decades. The ambition here is to learn more of the environment from data and reduce how much has to be built separately for every task.

A team could reconstruct a real setting and add a conventional physics engine, which calculates movement using rules engineers specify. The virtual objects still need suitable physical properties. A photograph doesn't tell you everything about mass or friction.

Or a model can learn those relationships from examples.

In DreamerV3, published in Nature in 2025, an AI learns a world model from interactions with an environment. It uses that model to generate sequences of predicted experience.

Imagine using that approach with our box. Inside the learned simulation, a push leaves it near the edge. The system estimates that completing the job from there will be harder. The part that chooses movements learns from that feedback across many imagined attempts.

This practice happens in internal numerical representations. There doesn't need to be a film to watch. Later, the system can act on what it learned.

I think this is easy to miss. We are used to judging generative AI by the thing it produces in front of us. A useful output here could be a skill that another machine acquired.

For our hypothetical packing team, the awkward box may appear too rarely to supply much practice, while appearing often enough to make deployment a nuisance.

If a simulation can reproduce that difficulty, the team could make it a regular part of training. The robot needn't wait for another inconvenient delivery.

This could change which jobs are worth automating. A task can be manageable most of the time and still be uneconomical once you include the effort of teaching the exceptions. Making useful practice easier to obtain could move that boundary.

The wrong kind of practice

In the 2018 World Models experiments, David Ha and Jürgen Schmidhuber trained a controller inside a learned version of a game where it had to avoid fireballs. In some simulated runs, it found movements that stopped monsters from shooting, or extinguished their fireballs as they appeared.

A fairly convenient universe to learn survival in.

Those tricks didn't work in the original game. The controller had learned to exploit errors in its training environment.

With a suitable amount of uncertainty added to the simulation, it did transfer successfully back to the original game. This wasn't physical-world transfer, but the result matters: an imperfect simulator could still teach useful behavior.

It needed to preserve enough of what made an action succeed or fail.

A packing simulation where every grip holds could teach the robot a routine that its real gripper cannot perform. More practice would make it better at the wrong task.

The check is whether that training helps on fresh physical attempts. World Labs reports hardware transfer and comparisons between simulated and physical behavior in its real-to-sim-to-real work. Those are results for particular tasks, not a guarantee for the next one.

Who chooses the experience?

Suppose our packing simulation does reproduce the problem well enough to help. Which situations should the robot encounter more often?

You could spend the compute changing the table's color. That might help if visual recognition is causing the failures. If a bent flap is the problem, all those colorful tables could leave the missing skill untouched.

Someone who knows the job may see that immediately. They may also recognize that two failures which look unrelated share the same cause. Their knowledge can help decide what the machine needs to practice.

That is a role I find more interesting than approving the next hundred AI outputs.

In Letter 94, I wrote about expensive people losing their week to reviewing machine output. Here, some of that judgment could shape the learning before the same correction reaches them again.

It changes what a record of a failure might be worth, too.

A video shows the team what happened. A useful reconstruction could let them change the conditions and investigate what would have prevented it. Would another approach work? Was the flap the problem, or the angle?

The simulation still has to get those consequences right. But if it does, one difficult encounter could supply practice for many variations the team hasn't met yet.

And if that practice teaches a skill that works outside the simulation, other machines with the right hardware and setup could use what was learned. Each one wouldn't have to wait for the same awkward box.

Someone still has to pick up the first fallen box. Capturing the failure and making it useful takes work as well.

What interests me is how far that work could travel.

The next machine might inherit the lesson without having to repeat the mistake.

Have a great weekend.

Stay sharp.

— Charafeddine (CM)