Layer 4 — Model Architecture · Chapter 2

The Building Blocks of Modern Models

Book chapterLayer 4

Modern models are assembled from recurring components. The names and details change, but the underlying functions are remarkably consistent: represent information, mix it, transform it, preserve useful signals, and generate an output.

Representations and embeddings

Raw inputs must be converted into numerical representations. Text is divided into tokens; images may be divided into patches; audio becomes frames or features; physical systems produce time-series measurements. An embedding maps each unit into a vector—a list of numbers that allows the model to compare, combine, and transform information.

Layers and nonlinear transformations

A neural network applies repeated mathematical transformations. Each layer receives a representation, changes it, and passes the result forward. Nonlinear operations allow the system to learn more than simple proportional relationships.

Information mixing

A model must decide which parts of an input should influence one another. Different architectures use different mixing mechanisms:

Residual connections and normalization

Deep models can become difficult to train because signals weaken, grow unstable, or lose useful detail. Residual connections let information bypass a transformation and be added back later. Normalization keeps activations within manageable ranges. These engineering ideas make very deep networks trainable.

Output heads

The final representation must be converted into something useful: a probability, a class, the next token, a bounding box, a control signal, or a predicted future state. The output head is tailored to that task even when the internal model is shared.

Dense and sparse computation

Dense model

Most or all parameters participate in every input. The design is simple, but computation grows with total model size.

Sparse model

Only selected components activate for a given input. This can increase total capacity without using every parameter each time.

The architecture of a model is less like one giant formula and more like a carefully organized network of reusable computational parts.