1 Transformer encoder fundamentals

1.1 Core purpose and typical inputs

A Transformer encoder transforms an input sequence into a sequence of context-rich vectors, where each element’s representation reflects information from other elements. The input can be textual tokens, image patches, or time steps from signals. During training or inference, the encoder computes a stack of learned transformations that progressively refine representations for downstream tasks.

1.2 Key architectural idea: self-attention

Self-attention lets the model compare every sequence element with every other element to decide which relationships matter. Unlike recurrence-based architectures that propagate information step by step, encoder self-attention directly connects distant positions, helping the model capture long-range dependencies. The mechanism is implemented with multiple attention heads that learn different interaction patterns in parallel.

1.3 Position information and sequence order

Because attention itself is permutation-invariant with respect to the sequence order, encoders inject position information so the model can distinguish, for example, “A follows B” from “B follows A.” This is commonly achieved with positional encodings or learned positional embeddings, which are combined with token or patch embeddings before attention layers.

2 Encoder layer structure

2.1 Multi-head self-attention sublayer

Each encoder layer typically begins with a multi-head self-attention module, producing updated representations that incorporate contextual mixing across the sequence.

2.1.1 Query, key, and value projections

For each attention head, the input vectors are projected into three learned spaces: queries, keys, and values. Queries represent what the current element is looking for, keys represent what each element offers, and values contain the content that gets aggregated. After projection, the model computes similarity scores between queries and keys to determine how much each element should contribute.

2.1.2 Attention weights and scaling

The similarity scores are normalized into attention weights, usually via a softmax. A scaling factor is applied to stabilize gradients when the dimensionality of keys is large. The final output for each head is a weighted sum of the corresponding value vectors, where higher weights correspond to stronger relevance.

2.1.3 Masking for selective attention

Masks allow selective attention over subsets of positions. For example, padding tokens in variable-length batches should not influence other elements, so an attention mask prevents them from contributing. In an encoder, attention is generally bidirectional (all non-masked positions are visible), unlike causal settings used for generation.

2.2 Feed-forward network (FFN) sublayer

Following self-attention, the encoder layer includes a position-wise feed-forward network that refines each position independently using the same learned transformation for all sequence elements.

2.2.1 Position-wise transformation

Although the FFN operates on each position separately, it changes feature dimensions to improve representational capacity. Because it is applied to every time step or token in parallel, it complements attention: attention mixes information across positions, while the FFN enhances nonlinear processing within each position’s channel space.

2.2.2 Activation functions and hidden expansion

The FFN typically consists of two linear layers with a nonlinear activation in between. A common design expands the hidden dimension to a larger intermediate size, applies an activation, then projects back to the model’s original dimensionality. This hidden expansion increases expressiveness without changing the sequence length.

2.3 Residual connections and normalization

Residual connections and normalization are essential for training deep networks reliably.

2.3.1 Skip/residual pathway rationale

A skip connection adds the input of a sublayer to its output, enabling gradients to flow more directly through the network. This helps optimization and reduces the risk that later layers completely overwrite earlier representations. Residual pathways also support stable learning when stacking many layers.

2.3.2 Layer normalization placement

Layer normalization is applied either before or after the sublayer computations, depending on the architectural variant. Proper placement affects training dynamics and gradient behavior, particularly in deep or large-batch regimes.

2.4 Dropout and regularization inside the encoder

Dropout and related techniques reduce overfitting and improve generalization.

2.4.1 Dropout usage patterns

Dropout can be applied to attention weights, to feed-forward activations, and to the output of sublayers before adding residual connections. The specific placement varies by implementation, but the goal is to prevent co-adaptation of features and reduce sensitivity to particular patterns in the training set.

2.4.2 Training stability considerations

Regularization must be balanced with optimization. Excessive dropout can slow convergence or undercut learning capacity, while too little may lead to overfitting. Many training setups tune dropout rates alongside learning-rate schedules and batch sizing.

3 Input representations and embeddings

3.1 Token embeddings and vocabulary mapping

For text inputs, tokens are mapped to vectors using an embedding table indexed by vocabulary IDs. These vectors are often scaled and then combined with positional information to provide both identity and order cues to the encoder.

3.2 Positional encoding vs learned position embeddings

Encoders require an ordering signal. Two broad approaches are used: fixed sinusoidal patterns and learned position vectors.

3.2.1 Sinusoidal positional encodings

Sinusoidal encodings use deterministic sine and cosine functions at different frequencies. They allow the model to generalize to unseen sequence lengths more easily because the encoding pattern continues analytically beyond the training range.

3.2.2 Learned positional embeddings

Learned positional embeddings treat each position index as a trainable vector. This approach can be simpler and sometimes yields strong performance, though it may require careful handling for sequences longer than those observed during training.

3.3 Segment/type embeddings and multimodal variants

When inputs contain multiple parts or modalities, additional embeddings may be introduced.

3.3.1 Segment embeddings in paired inputs

For paired sequences, such as two text spans, segment embeddings indicate which tokens belong to which part. This helps the encoder learn interactions across segments while maintaining awareness of token provenance.

3.3.2 Patch embeddings for vision use cases

For images, inputs are partitioned into patches, flattened, and linearly projected into embedding vectors. These patch embeddings replace token embeddings, and positional encodings are adapted to the patch grid structure.

4 Architectural variations of the encoder

4.1 Pre-norm vs post-norm Transformer encoders

Normalization placement differentiates two common variants: pre-norm and post-norm.

4.1.1 Implications for gradient flow

In pre-norm designs, layer normalization is applied before the attention or FFN sublayer, which often improves gradient flow in deep networks. Post-norm applies normalization after residual addition, which can be effective but may require more careful optimization settings.

4.1.2 Common training behaviors

Pre-norm encoders are widely used in modern large-scale training because they tend to be more stable across depth. Post-norm variants may appear in earlier designs or in specialized setups where optimization behavior is well-characterized.

4.2 Relative and rotary positional methods

Some methods modify how position influences attention scores rather than simply adding a position embedding to the input.

4.2.1 Relative positional bias

Relative methods incorporate information about the distance or relationship between positions. Relative positional bias can be added to attention logits, allowing the model to learn how relative offsets affect relevance.

4.2.2 Rotary position embeddings (RoPE)

RoPE rotates query and key vectors in a way that encodes positional relationships within the attention computation. This approach has gained popularity for its ability to extend to longer contexts with appropriate tuning, while preserving a structured way of embedding order into attention.

4.3 Efficient attention mechanisms

Standard attention can be costly for long sequences because it scales quadratically with sequence length. Efficient attention aims to reduce compute or memory requirements.

4.3.1 Sparse and local attention

Sparse or windowed attention limits which token pairs interact, such as restricting attention to nearby positions or predetermined patterns. This reduces complexity while still capturing local structure and some broader context via layered expansion.

4.3.2 Linearized attention approaches

Linearized methods restructure attention computations to reduce dependence on sequence length in the dominant term. These approaches approximate or transform attention so it can be computed with lower complexity, often at the cost of different inductive biases.

4.3.3 Memory-efficient attention patterns

Implementations may also reduce peak memory through algorithmic tricks, such as recomputation strategies or fused kernels. While the exact mathematical attention remains similar, practical efficiency can significantly improve throughput for large models.

5 Practical training considerations

5.1 Attention masks and padding handling

Training typically batches examples of different lengths, requiring padding. Attention masks ensure the encoder does not use padded positions when computing context.

5.1.1 Padding masks for variable-length batches

A padding mask marks which tokens are real and which are filler. Masking prevents attention from assigning weight to padded tokens and avoids contaminating learned representations with meaningless content.

5.1.2 Causal vs bidirectional attention in encoders

Encoders are normally bidirectional: tokens can attend to future and past tokens. Causal masking is primarily used in decoder components for autoregressive generation, but variants may repurpose encoder-like blocks for constrained attention patterns.

5.2 Hyperparameters that affect performance

Model quality depends on architecture choices and optimization settings.

5.2.1 Number of layers and hidden size

More layers can increase representational depth, while a larger hidden size increases capacity per position. However, increasing both often raises compute cost and can require adjusting regularization and learning-rate policies.

5.2.2 Number of attention heads

Multiple heads allow attention to focus on diverse relation types. Too few heads can limit interaction diversity, while too many can make each head’s subspace too narrow unless the model width and training regime are scaled accordingly.

5.2.3 Learning rate schedules and warmup

Learning-rate schedules control how quickly the model adapts during training. Warmup phases often help avoid unstable updates early on, especially for large batches or deep networks. Later decay phases help converge to better minima.

5.3 Fine-tuning strategies for downstream tasks

Encoders are commonly adapted to task-specific objectives using fine-tuning.

5.3.1 Feature extraction vs full fine-tuning

Feature extraction freezes encoder weights and trains only a lightweight head, which is useful when compute is limited or labeled data is scarce. Full fine-tuning updates all encoder parameters, usually improving task performance but requiring more careful training to avoid overfitting.

5.3.2 Layer freezing and differential learning rates

Intermediate strategies include freezing early layers, which often capture general patterns, while fine-tuning later layers that are more task-specific. Differential learning rates can assign smaller rates to pretrained layers and larger rates to newly initialized components.

6 Outputs and downstream use

6.1 Sequence representations (token-level embeddings)

The encoder outputs a vector for each input position. These token-level representations serve tasks such as tagging, span detection, or any setting where predictions depend on local context informed by the entire sequence.

6.2 Pooled representations (sentence-level vectors)

To obtain a single vector per sequence, models apply pooling strategies such as selecting a designated special token, averaging over token embeddings, or using learned pooling layers. The pooled representation is convenient for tasks where the target is global to the whole input.

6.3 Task heads commonly attached to encoders

Downstream models attach lightweight prediction modules to the encoder outputs.

6.3.1 Classification heads

A classification head maps either a pooled vector or a particular token representation to logits over labels. Training typically uses a classification loss aligned with the label format, such as cross-entropy.

6.3.2 Token labeling heads

For token-level tasks, a labeling head applies a linear projection (possibly with activation) to each token representation. Loss functions then compare predicted tags to ground-truth labels for each position.

6.3.3 Contrastive and retrieval-oriented heads

Retrieval-oriented setups often use projection heads to embed inputs into a shared space. Similarity measures (dot products or normalized cosine similarity) support objectives like contrastive learning, where correct pairs are encouraged to be close and incorrect pairs separated.

7 Computational aspects and scaling

7.1 Complexity drivers (sequence length and heads)

The dominant cost of full self-attention arises from computing similarity between token pairs, leading to quadratic dependence on sequence length. Number of heads increases the cost modestly because the same total model width is split across heads, but implementation details can affect efficiency.

7.2 Memory usage and batching strategies

Memory consumption includes storing activations for backpropagation and the attention matrices. Techniques such as gradient checkpointing, reduced precision (e.g., mixed precision), and careful batch sizing help fit models into available hardware. Padding efficiency also matters: minimizing padded tokens improves effective batch utilization.

7.3 Scaling laws intuition for encoder depth/width

Scaling intuition suggests that increasing model capacity (depth and width) and training compute can improve performance, often with diminishing returns at higher scales. Practical scaling must consider optimization stability and regularization, since naive increases in size can degrade generalization without corresponding training adjustments.

8 Evaluation and diagnostics

8.1 Measuring attention behavior

Attention diagnostics examine how strongly the model links positions. Metrics can include attention entropy, sparsity patterns (for masked or efficient attention variants), and visualization of attention maps for interpretability studies. These tools help detect whether the model attends to relevant spans or collapses to trivial patterns.

8.2 Probing encoder representations

Probing assesses whether specific linguistic or structural properties are present in intermediate layers. A common approach trains lightweight classifiers on frozen encoder states to predict properties such as part-of-speech tags or syntactic features. Care must be taken to ensure probes do not simply memorize surface cues.

8.3 Common failure modes (overfitting, underfitting, collapse)

Overfitting can appear as strong training performance with degraded validation accuracy, sometimes accompanied by excessive confidence. Underfitting may present as uniformly mediocre results and low sensitivity to learning signals. Representation collapse can be detected when embeddings become too similar across inputs, reducing effective diversity; monitoring embedding norms and similarity distributions can help diagnose this issue.

9.1 Encoder-only vs encoder–decoder Transformers

Encoder-only Transformers use the encoder stack for understanding and representation learning, while encoder–decoder Transformers add a decoder stack for sequence generation. In encoder–decoder designs, cross-attention enables the decoder to attend to encoder outputs, supporting tasks like translation or summarization.

9.2 Notable encoder-based model families (generic overview)

Many modern architectures build on the encoder concept, differing in positional encoding, attention efficiency, normalization strategy, and training objectives. Families are often distinguished by their pretraining tasks and the choices made to handle longer contexts or multimodal inputs.

9.3 Tokenization choices that pair with encoders

Tokenization affects the granularity of the encoder’s input representations. Common schemes include subword tokenization, which balances vocabulary size with the ability to represent rare words. For multimodal variants, tokenization extends to patching strategies and modality-specific feature discretization.

10 Pseudocode and reference implementation sketch

10.1 Minimal forward pass outline

A minimal encoder forward pass computes token embeddings, adds positional information, then repeatedly applies attention and feed-forward transformations with residual connections and normalization. The result is either the full sequence of vectors or pooled outputs used by a task head.

10.2 Shape conventions for inputs and outputs

Let the batch size be \(B\), sequence length be \(L\), and model hidden size be \(d\). Inputs are typically shaped as \( (B, L) \) for token IDs and \( (B, L, d) \) after embedding. Attention outputs preserve the sequence length, yielding \( (B, L, d) \) at each layer. Pooling reduces to \( (B, d) \) when producing sentence-level vectors.

10.3 Typical training loop structure

A typical training loop samples minibatches, constructs attention masks for padded positions, runs the encoder to obtain outputs, computes a task loss via the attached head, and performs backpropagation followed by an optimizer step. Learning-rate scheduling updates the optimizer’s learning rate each iteration or epoch, while regularization mechanisms such as dropout operate during training but are disabled during evaluation.