Language model
Predicts and transforms symbolic sequences, often without direct physical feedback.
Layer 4 — Model Architecture · Chapter 6
Human intelligence combines language, vision, sound, touch, movement, time, and an internal sense of the surrounding world. Multimodal architectures attempt to connect several forms of information inside one system.
A system must learn that a spoken name, a written word, a photograph, and a physical object can refer to the same thing. Alignment creates shared representations, but fine-grained understanding also requires preserving what is unique to each modality: spatial layout, timing, pitch, motion, depth, or force.
Multimodal models may interpret an image, describe a video, produce speech, generate pictures, control a robot, or predict future frames. Perception and generation reinforce each other because a system that can model how data is produced may also become better at interpreting it.
A world model represents how an environment changes. It may predict the next visual frame, the consequences of an action, the motion of an object, the state of a game, or the behavior of a physical system. Planning can then occur inside the model before an action is attempted in reality.
Robots add a closed loop: sense the world, update an internal state, select an action, move, observe the result, and adjust. Real-time control introduces constraints that language-only systems can often ignore—latency, safety, geometry, uncertainty, wear, energy, and irreversible mistakes.
Predicts and transforms symbolic sequences, often without direct physical feedback.
Predicts how a state evolves, including the likely consequences of actions.
A model becomes more than multimodal when its different senses contribute to one coherent, predictive understanding of the world.