Dense model
Most or all parameters participate in every input. The design is simple, but computation grows with total model size.
Layer 4 — Model Architecture · Chapter 2
Modern models are assembled from recurring components. The names and details change, but the underlying functions are remarkably consistent: represent information, mix it, transform it, preserve useful signals, and generate an output.
Raw inputs must be converted into numerical representations. Text is divided into tokens; images may be divided into patches; audio becomes frames or features; physical systems produce time-series measurements. An embedding maps each unit into a vector—a list of numbers that allows the model to compare, combine, and transform information.
A neural network applies repeated mathematical transformations. Each layer receives a representation, changes it, and passes the result forward. Nonlinear operations allow the system to learn more than simple proportional relationships.
A model must decide which parts of an input should influence one another. Different architectures use different mixing mechanisms:
Deep models can become difficult to train because signals weaken, grow unstable, or lose useful detail. Residual connections let information bypass a transformation and be added back later. Normalization keeps activations within manageable ranges. These engineering ideas make very deep networks trainable.
The final representation must be converted into something useful: a probability, a class, the next token, a bounding box, a control signal, or a predicted future state. The output head is tailored to that task even when the internal model is shared.
Most or all parameters participate in every input. The design is simple, but computation grows with total model size.
Only selected components activate for a given input. This can increase total capacity without using every parameter each time.
The architecture of a model is less like one giant formula and more like a carefully organized network of reusable computational parts.