Layer 4 — Model Architecture · Chapter 6

Multimodal Systems and World Models

Book chapterLayer 4

Human intelligence combines language, vision, sound, touch, movement, time, and an internal sense of the surrounding world. Multimodal architectures attempt to connect several forms of information inside one system.

Three broad approaches to multimodality

  1. Separate encoders with a shared interface. Specialized components convert text, images, audio, or sensor data into compatible representations.
  2. Unified tokenization. Different modalities are expressed as sequences that a common model can process.
  3. Jointly trained architectures. The system learns cross-modal relationships from the beginning rather than attaching one modality later.

Alignment between modalities

A system must learn that a spoken name, a written word, a photograph, and a physical object can refer to the same thing. Alignment creates shared representations, but fine-grained understanding also requires preserving what is unique to each modality: spatial layout, timing, pitch, motion, depth, or force.

Generation and perception

Multimodal models may interpret an image, describe a video, produce speech, generate pictures, control a robot, or predict future frames. Perception and generation reinforce each other because a system that can model how data is produced may also become better at interpreting it.

World models

A world model represents how an environment changes. It may predict the next visual frame, the consequences of an action, the motion of an object, the state of a game, or the behavior of a physical system. Planning can then occur inside the model before an action is attempted in reality.

Embodied architectures

Robots add a closed loop: sense the world, update an internal state, select an action, move, observe the result, and adjust. Real-time control introduces constraints that language-only systems can often ignore—latency, safety, geometry, uncertainty, wear, energy, and irreversible mistakes.

Language model

Predicts and transforms symbolic sequences, often without direct physical feedback.

World model

Predicts how a state evolves, including the likely consequences of actions.

Open challenges

A model becomes more than multimodal when its different senses contribute to one coherent, predictive understanding of the world.