1 Overview and Motivation
Capsule networks (CapsNets) are a class of artificial neural network architectures introduced by Geoffrey Hinton and his colleagues in 2017. Unlike traditional convolutional neural networks (CNNs), which process spatial features through pooling layers that discard positional information, capsule networks preserve hierarchical spatial relationships by grouping neurons into "capsules"—vectors that encode both the probability of an entity's presence and its instantiation parameters (e.g., pose, orientation, scale). CapsNets use dynamic routing or routing-by-agreement mechanisms between layers to ensure that capsules in lower layers send their outputs only to compatible higher‑level capsules, thereby achieving translation equivariance and better robustness to viewpoint changes. They are particularly studied in computer vision tasks such as image classification, segmentation, and generative modeling.
1.1 Limitations of Convolutional Neural Networks
1.1.1 Pooling information loss
In standard CNNs, pooling layers (e.g., max pooling) downsample feature maps by retaining only the maximum activation in each local region. This operation discards precise spatial information such as the relative positions of detected features. While pooling improves translational invariance and reduces computational load, it can hinder the network's ability to model fine‑grained spatial hierarchies. Consequently, CNNs may confuse objects whose parts have the same local features but different arrangements.
1.1.2 Lack of viewpoint invariance
CNNs can learn viewpoint‑invariant representations to some degree through data augmentation, but they do not explicitly encode relationships between parts and wholes. A CNN trained on frontal views of an object may fail to recognize the same object from a novel viewpoint if it has not seen similar rotated or scaled examples during training. The lack of built‑in equivariance to affine transformations (such as rotation or scaling) means that the network must dedicate capacity to learn many redundant filters for each orientation.
1.2 The Capsule Idea
1.2.1 Vector neurons vs. scalar neurons
In traditional neural networks, each neuron outputs a single scalar value representing the probability or activation of a learned feature. Capsule networks replace these scalar neurons with vector neurons: each capsule outputs a vector whose magnitude encodes the probability of the entity being present and whose orientation represents its instantiation parameters. This vector representation allows the capsule to capture not just *whether* a feature exists but also *how* it is instantiated (e.g., its exact position, rotation, scale, and other properties).
1.2.2 Instantiation parameters and equivariance
The instantiation parameters stored in a capsule vector are designed to be equivariant to transformations of the input. For example, if an object in the input image rotates, the vector of the capsule representing that object should rotate correspondingly. This equivariance is achieved through the routing mechanism, which ensures that lower‑level parts (e.g., edges) are correctly grouped to form higher‑level wholes (e.g., objects) while preserving the spatial relationships among the parts. Unlike the translation‑invariant pooling in CNNs, capsule networks aim for equivariance, enabling the model to generalize more robustly to novel viewpoints.
2 Architecture of Capsule Networks
2.1 Primary Capsule Layer
2.1.1 Convolutional filter transformation
The primary capsule layer receives feature maps from a standard convolutional layer. Each feature map is reshaped into a set of capsules. Specifically, a convolutional operation with multiple filters produces a tensor of shape (height, width, channels). These channels are then grouped to form capsule vectors. For instance, if the convolutional layer outputs K filter maps, each of size H × W, the primary capsule layer produces H × W capsules, each of dimension d = K / C (where C is the number of capsule types). The transformation is learned via standard backpropagation.
2.1.2 Squashing function
The output vectors of the primary capsule layer are passed through a nonlinear squashing function that shrinks short vectors to near zero and saturates long vectors to near unit length. The squashing function is defined as:
\[
| \mathbf{v}_j = \frac{\|\mathbf{s}_j\|^2}{1 + \|\mathbf{s}_j\|^2} \frac{\mathbf{s}_j}{\|\mathbf{s}_j\|} |
|---|
\]
where \(\mathbf{s}_j\) is the total input to capsule \(j\) and \(\mathbf{v}_j\) is its output. This non‑linearity ensures that the vector length (interpreted as the probability of entity presence) is bounded between 0 and 1, while the direction remains meaningful.
2.2 Higher‑Level Capsule Layer
2.2.1 Prediction vectors
| For a capsule \(i\) in layer \(L\) and a capsule \(j\) in layer \(L+1\), a prediction vector \(\hat{\mathbf{u}}_{j | i}\) is computed by multiplying the output \(\mathbf{u}_i\) of capsule \(i\) by a learned weight matrix \(\mathbf{W}_{ij}\): |
|---|
\[
| \hat{\mathbf{u}}_{j | i} = \mathbf{W}_{ij} \mathbf{u}_i |
|---|
\]
This weight matrix captures the spatial and part‑whole relationship between the two capsules. The prediction vector represents how capsule \(i\) "votes" for the instantiation of capsule \(j\).
2.2.2 Coupling coefficients
The coupling coefficients \(c_{ij}\) determine how much of the output from lower‑layer capsule \(i\) is routed to higher‑layer capsule \(j\). These coefficients are non‑negative and sum to 1 over all higher‑layer capsules \(j\) for each lower‑layer capsule \(i\). They are computed using a softmax function over the routing logits \(b_{ij}\):
\[ c_{ij} = \frac{\exp(b_{ij})}{\sum_k \exp(b_{ik})} \]
Initially, the logits \(b_{ij}\) are set to zero, implying uniform coupling. They are then iteratively updated during the dynamic routing procedure.
2.3 Dynamic Routing Algorithm
2.3.1 Agreement computation
| After computing the weighted sum of prediction vectors \(\mathbf{s}_j = \sum_i c_{ij} \hat{\mathbf{u}}_{j | i}\), the agreement between the actual output of capsule \(j\) (after squashing) and the predictions from lower capsules is measured by the dot product: |
|---|
\[
| a_{ij} = \mathbf{v}_j \cdot \hat{\mathbf{u}}_{j | i} |
|---|
\]
A high positive agreement indicates that capsule \(i\)'s prediction aligns well with the activation of capsule \(j\), suggesting that capsule \(i\) is part of the entity represented by capsule \(j\).
2.3.2 Iterative routing steps
The routing logits \(b_{ij}\) are updated by adding the agreement \(a_{ij}\) to them:
\[ b_{ij} \leftarrow b_{ij} + a_{ij} \]
This update increases the coupling coefficient \(c_{ij}\) for agreements and decreases it for disagreements. The entire process (computing \(c_{ij}\), \(\mathbf{s}_j\), \(\mathbf{v}_j\), and \(a_{ij}\)) is repeated for a fixed number of iterations (typically 3). The iterative routing refines the coupling, allowing the network to selectively propagate information from lower capsules to the most compatible higher capsules.
3 Variants and Extensions
3.1 Matrix Capsules with EM Routing
3.1.1 Pose matrices
Instead of using vectors, matrix capsules represent each entity with a pose matrix and a presence probability. The pose matrix (typically 4×4) encodes the spatial transformation (e.g., rotation, translation, scaling) of the entity relative to a canonical coordinate frame. This representation enables more explicit geometric reasoning.
3.1.2 Expectation‑maximization approach
Routing in matrix capsules is performed using an expectation‑maximization (EM) algorithm. In the E‑step, the presence probabilities of higher‑level capsules are estimated based on the votes from lower‑level capsules. In the M‑step, the pose parameters of higher‑level capsules are updated to maximize the likelihood of the observed lower‑level activations. EM routing provides a principled probabilistic interpretation and can be more computationally efficient than dynamic routing with vectors.
3.2 Stacked Capsule Autoencoders
3.2.1 Capsule decoder
Stacked Capsule Autoencoders (SCAE) extend capsule networks to unsupervised learning. The model consists of a part encoder and a capsule decoder. The decoder takes the activations of learned capsules and reconstructs the input image by assembling parts (e.g., small image patches) according to the capsules' predicted poses and presence.
3.2.2 Part‑whole relationship learning
SCAE explicitly learns part‑whole hierarchies without requiring labels. Lower‑level parts are grouped into objects by capsule layers, and the autoencoder framework encourages the capsules to capture disentangled factors of variation. This approach has shown promise for unsupervised learning of object‑centric representations.
3.3 Other Routing Mechanisms
3.3.1 Attention‑based routing
Several works have replaced the dynamic routing algorithm with attention mechanisms. For example, capsule networks can use self‑attention to compute coupling coefficients, or cross‑attention between layers. Attention‑based routing can be more flexible and may integrate better with transformer architectures.
3.3.2 Convolutional capsule networks
Convolutional capsule networks (CapsCNNs) apply convolutional operations over capsules. Instead of having a fully connected matrix transformation between capsule layers, they use local receptive fields and shared weight kernels. This design reduces the number of parameters and makes capsule networks more scalable to high‑resolution images while preserving part‑whole relationships.
4 Applications in Computer Vision
4.1 Image Classification
4.1.1 MNIST and small‑scale benchmarks
Capsule networks achieved state‑of‑the‑art performance on MNIST, demonstrating high accuracy even on highly overlapping digits. The ability to model spatial hierarchies makes them particularly robust to affine transformations. On small‑scale datasets (e.g., CIFAR‑10), capsule networks have shown competitive results, often outperforming CNNs with fewer parameters.
4.1.2 Large‑scale challenges (e.g., ImageNet)
Scaling capsule networks to large‑scale datasets like ImageNet remains challenging due to the high computational cost of routing and the difficulty of training deep capsule architectures. However, later variants (e.g., convolutional capsule networks and matrix capsules with EM routing) have made progress, achieving competitive accuracy on subsets or with appropriate modifications.
4.2 Object Detection and Segmentation
4.2.1 Instance segmentation with capsules
Capsule networks can be extended to instance segmentation by assigning each pixel to the capsule that best represents the object it belongs to. The routing mechanism naturally groups pixels into parts and parts into objects, providing a strong inductive bias for segmentation tasks. Results on small‑scale datasets have shown that capsules can produce high‑quality segmentation masks.
4.2.2 Multi‑object recognition
Because capsule networks can represent multiple entities in a single image through separate capsules (each with its own probability), they are well‑suited for multi‑object recognition. The dynamic routing ensures that different objects are assigned to different capsules, reducing confusion between overlapping instances compared to traditional CNNs that pool spatially.
4.3 Generative Models
4.3.1 Capsule‑based variational autoencoders
Capsule networks have been integrated into variational autoencoders (VAEs) to learn disentangled representations. The capsule vectors can directly encode factors such as object pose, scale, and identity. The VAE framework provides a probabilistic encoder (the capsule network) and a decoder that reconstructs the image from the capsule activations.
4.3.2 Image reconstruction and disentangled representation
By training a capsule network with a reconstruction loss, the model learns to represent objects in a disentangled manner. For example, it can generate images of rotated or rescaled objects by manipulating the corresponding dimensions of the capsule vectors. This capability is valuable for tasks requiring explicit control over generative factors.
5 Training and Implementation Considerations
5.1 Loss Functions
5.1.1 Margin loss for classification
The standard loss for capsule network classification is a margin loss applied to each output capsule. For capsule \(k\), the loss is:
\[
| L_k = T_k \max(0, m^+ - \|\mathbf{v}_k\|)^2 + \lambda (1 - T_k) \max(0, \|\mathbf{v}_k\| - m^-)^2 |
|---|
\]
where \(T_k = 1\) if the entity (class) is present, and \(T_k = 0\) otherwise. Typical margin values are \(m^+ = 0.9\), \(m^- = 0.1\), and \(\lambda = 0.5\). This loss encourages the correct capsule to have a long output vector (close to 1) and the incorrect capsules to have short vectors (close to 0).
5.1.2 Reconstruction loss as regularization
A decoder network is often attached to the capsules to reconstruct the input image. The reconstruction loss (e.g., mean squared error) acts as a regularizer, forcing the capsule vectors to encode enough information to reconstruct the image accurately. This regularization improves the quality of the capsule representations and often boosts classification accuracy.
5.2 Computational Efficiency
5.2.1 Routing iteration cost
The dynamic routing algorithm requires multiple iterations (typically 3) per forward pass, each involving matrix multiplication, softmax, squashing, and dot products. This iterative process can be several times more expensive than a forward pass through a comparable CNN layer. The cost scales with the number of capsules and the capsule dimension.
5.2.2 Comparison to CNNs
For small‑scale tasks, capsule networks can be more parameter‑efficient than CNNs because they reuse weight matrices across capsules. However, for large‑scale tasks, the routing cost becomes a significant bottleneck. Convolutional capsule networks and EM routing reduce this overhead, but capsule networks generally remain slower than standard CNNs of similar depth.
5.3 Open‑Source Frameworks and Libraries
5.3.1 TensorFlow and PyTorch implementations
The original capsule network paper provided an open‑source implementation in TensorFlow. Since then, numerous PyTorch implementations have been released, often with improved modularity and support for GPU acceleration. Both frameworks offer convenient APIs for building custom capsule layers.
5.3.2 Notable repositories
Several notable repositories exist on GitHub, including "CapsNet‑TensorFlow" by the original authors, "CapsNet‑PyTorch" by community contributors, and "Matrix‑Capsules‑EM‑TensorFlow" for matrix capsule variants. These repositories serve as reference implementations and often include pretrained models on MNIST and other small datasets.
6 Current Trends and Open Challenges
6.1 Scalability to High‑Resolution Images
One of the primary open challenges is scaling capsule networks to high‑resolution images (e.g., 224×224 as in ImageNet). The number of capsules grows quadratically with image dimensions, making the routing computation prohibitively expensive. Research continues on efficient routing algorithms and hierarchical capsule structures that can handle larger spatial sizes.
6.2 Integration with Transformers and Attention Mechanisms
Recent work explores combining capsule networks with transformer architectures. For example, using attention to compute coupling coefficients or replacing the squashing function with more flexible normalization. These hybrid approaches aim to leverage the strengths of both paradigms: the part‑whole reasoning of capsules and the scalability of transformers.
6.3 Theoretical Understanding of Routing
The dynamic routing algorithm lacks a rigorous theoretical justification compared to backpropagation in standard networks. Understanding why routing improves generalization and how it relates to graphical models or Bayesian inference is an active research area. New insights may lead to simpler and more effective routing mechanisms.
6.4 Relationship to Other Equivariant Networks
Capsule networks are part of a broader family of equivariant neural networks, including group equivariant CNNs and steerable CNNs. Clarifying the connections and relative advantages of these approaches is an ongoing topic. Capsule networks offer a more direct representation of part‑whole hierarchies, while group equivariant methods provide exact equivariance to predefined transformation groups. Combining these ideas may yield powerful architectures.