few-shot learning

1.1 Definition and Key Concepts

Few-shot learning is a subfield of machine learning that focuses on enabling models to learn effectively from a very small number of labeled examples per class, typically between one and five. The core concept is to train a model that can generalize to new tasks with minimal supervision, leveraging prior knowledge acquired from related tasks. Key concepts include the *support set* (the few labeled examples for a new task) and the *query set* (unlabeled examples to be classified). The model is expected to perform well on the query set after being exposed only to the support set.

1.2 Motivation and Importance

Traditional supervised learning requires large, annotated datasets, which are often expensive or impractical to obtain in many real-world domains. Few-shot learning addresses this limitation by mimicking human ability to learn new concepts from just a few examples. Its importance spans applications such as medical imaging (where labeled data is scarce), personalized recommendations, and rare species recognition. By reducing data requirements, few-shot learning lowers the barrier for deploying machine learning in data-limited scenarios.

1.3 Relationship to Meta‑Learning and Transfer Learning

Few-shot learning is closely related to meta-learning (learning to learn) and transfer learning. Meta-learning treats each few-shot task as a training instance, learning a prior that facilitates rapid adaptation to new tasks. Transfer learning typically pretrains a model on a large source dataset and then fine-tunes on a target task; few-shot learning extends this by requiring adaptation from very few examples, often without fine-tuning the entire model. While transfer learning focuses on cross-domain knowledge, meta-learning emphasizes cross-task generalization, with few-shot learning providing a natural benchmark for both.

2.1 N‑Way K‑Shot Classification

In N‑way K‑shot classification, a task consists of N distinct classes, each with K labeled examples (the support set). The model must classify unseen examples (the query set) into one of these N classes. Typical settings include N=5 or N=20 and K=1 (one-shot) or K=5 (five-shot). This formulation enables controlled evaluation of a model’s ability to learn from minimal data.

2.2 Support Set and Query Set

The support set contains the K labeled examples per class that define the task. The query set contains additional unlabeled examples from the same N classes. During evaluation, the model uses the support set to infer class distinctions and then predicts labels for the query set. The performance metric is usually accuracy averaged over many randomly sampled episodes.

2.3 Episodic Training Strategy

Episodic training structures the meta-learning process by repeatedly sampling tasks (episodes) from a base dataset. Each episode consists of a support set and a query set. The model is updated to minimize classification error on the query set given the support set. This mimics the test-time scenario and forces the model to learn a generalizable learning procedure rather than memorizing fixed class labels.

3.1 Metric‑Based Methods

Metric-based methods learn an embedding space where examples from the same class are close and examples from different classes are distant. Classification is performed by comparing the query embedding to support set prototypes.

3.1.1 Siamese Networks

Siamese networks use twin subnetworks with shared weights to compute a similarity score between two inputs. During training, pairs of examples are presented, and the network learns to output high similarity for same-class pairs and low similarity for different-class pairs. At test time, a query is compared to each support example, and the class of the closest match is assigned.

3.1.2 Prototypical Networks

Prototypical networks compute a prototype vector for each class by averaging the embeddings of the support examples. A query is classified by finding the nearest prototype in the embedding space using a distance metric (typically Euclidean). This method is simple, effective, and widely used.

3.1.3 Matching Networks

Matching networks use an attention mechanism over the support set to compute a weighted nearest-neighbor classifier. The query embedding is compared to each support example, and the attention weights determine the predicted class. The architecture supports both full-context embedding (using a bidirectional LSTM over the support set) and a simple embedding network.

3.1.4 Relation Networks

Relation networks learn a non-linear comparison function between a query and support examples. Instead of using a fixed distance metric, a neural network (relation module) takes concatenated embeddings of a query and a support example and outputs a relation score. The query is assigned to the class with the highest aggregated relation score.

3.2 Model‑Based Methods

Model-based methods design architectures that can rapidly encode new information in their internal state, often using memory or fast weights.

3.2.1 Memory‑Augmented Neural Networks

Memory-augmented neural networks (MANNs) extend neural networks with external memory that can be read from and written to. In the few-shot setting, the support examples are stored in memory, and the model uses attention to retrieve relevant information when classifying queries. Neural Turing Machines and Differentiable Neural Computers are classic examples.

3.2.2 Meta Networks

Meta Networks (MetaNet) introduce a separate fast weight network that generates task-specific parameters on the fly. The meta-learner produces weight updates for the base learner based on the support set. This allows the model to adapt quickly without iterative gradient steps.

3.3 Optimization‑Based Methods

Optimization-based methods treat meta-learning as learning a good initialization or optimizer that enables rapid fine-tuning with few gradient steps.

3.3.1 Model‑Agnostic Meta‑Learning (MAML)

MAML learns an initial set of model parameters such that a small number of gradient steps on a few examples from a new task yields strong performance. During meta-training, the model is updated to minimize the loss after one or more gradient steps on the support set. MAML is model-agnostic and works with any differentiable architecture.

3.3.1.1 First‑Order MAML (FOMAML)

FOMAML simplifies MAML by ignoring second-order derivatives when computing the meta-gradient. Instead, it approximates the gradient using only first-order terms, reducing computational cost while maintaining competitive performance. This approximation is often sufficient in practice.

3.3.1.2 Reptile

Reptile is a first-order meta-learning algorithm that performs stochastic gradient descent on tasks, moving the initialization toward the average of task-specific fine-tuned parameters. Unlike MAML, Reptile does not explicitly compute gradient through the inner loop, making it simpler and more scalable.

3.3.2 LSTM Meta‑Learner

The LSTM meta-learner trains a separate recurrent neural network (LSTM) to act as an optimizer for the base model. The LSTM takes gradients as input and outputs parameter updates. By learning the update rule across many tasks, the LSTM can generalize to new tasks and adapt quickly with few steps.

4.1 Omniglot

Omniglot consists of 1,623 handwritten characters from 50 different alphabets, each with 20 examples. It is a standard benchmark for few-shot learning, often with 20-way 1-shot or 5-shot tasks. The large number of classes and simple image format make it suitable for rapid experimentation.

4.2 MiniImageNet

MiniImageNet is a subset of ImageNet containing 100 classes with 600 images each. It is divided into 64 training, 16 validation, and 20 test classes. MiniImageNet is more challenging than Omniglot due to higher image complexity and resolution, and it has become a de facto benchmark for few-shot image classification.

4.3 TieredImageNet

TieredImageNet is another ImageNet subset designed to reduce class correlation by partitioning classes into supercategories. It has 608 classes (20 supercategories) split into training, validation, and test sets. This structure tests generalization to novel classes that are more distinct from training classes.

4.4 Few‑Shot Language Understanding Benchmarks

For natural language processing, benchmarks such as FewRel (relation extraction), Cross-lingual NLU, and various text classification datasets (e.g., Amazon reviews, 20 Newsgroups) have been adapted for few-shot evaluation. These benchmarks measure a model’s ability to learn from a handful of text examples.

5.1 Computer Vision

5.1.1 Image Classification

Few-shot image classification is the most common application, enabling recognition of new object categories from a few photos. It is used in domains like wildlife identification, product cataloging, and personal photo organization where labeling large datasets is impractical.

5.1.2 Object Detection and Segmentation

Few-shot object detection and segmentation extend the concept to locating and delineating objects. Models learn to detect novel objects from a few bounding boxes or segmentation masks. Applications include medical image analysis (e.g., detecting rare tumors) and robotic grasping of unfamiliar objects.

5.2 Natural Language Processing

5.2.1 Text Classification

Few-shot text classification allows categorization of documents or sentences into new topics with minimal labeled data. Use cases include spam filtering for emerging scams, intent detection in chatbots, and sentiment analysis for niche product reviews.

5.2.2 Relation Extraction

Few-shot relation extraction trains models to identify semantic relationships between entities in text from a few labeled examples. It is valuable for building knowledge bases from scientific literature or news articles where new relation types frequently appear.

5.3 Robotics and Control

In robotics, few-shot learning enables a robot to learn new manipulation skills or navigation policies from a handful of demonstrations. For example, a robot can adapt to a new object’s shape or a new terrain type after observing only a few successful trajectories. This reduces the need for extensive manual programming.

5.4 Drug Discovery and Material Science

Few-shot learning accelerates discovery in chemistry and materials science by predicting properties of novel compounds from limited experimental data. For instance, predicting a molecule’s binding affinity or a material’s band gap from just a few measurements can guide targeted synthesis and screening.

6.1 Domain Shift and Distribution Mismatch

Few-shot models often assume that meta-training and meta-test tasks come from similar distributions. When the test domain differs significantly (e.g., medical images vs. natural images), performance degrades. Domain shift remains a major hurdle for real-world deployment.

6.2 Scalability to Larger Numbers of Classes

Most few-shot methods are designed for small N (e.g., 5 or 20). Scaling to hundreds or thousands of classes with only a few examples per class poses computational and algorithmic challenges, as nearest-neighbor searches become expensive and prototypes may not be discriminative.

6.3 Overfitting and Model Capacity

With extremely limited data (especially 1-shot), models easily overfit to the support set. Balancing model capacity to capture useful priors without memorizing training tasks is an ongoing issue. Regularization and data augmentation techniques are often necessary.

6.4 Evaluation Protocol Standardization

Different studies use varying evaluation protocols (e.g., number of episodes, way and shot settings, sample selection strategies). This makes fair comparison difficult. Efforts such as the FewShot CIFAR100 benchmark and standardized splits for MiniImageNet have improved consistency, but cross-paper comparisons remain challenging.

7.1 Integration with Self‑Supervised Learning

Self-supervised learning (e.g., contrastive learning) can learn rich representations without labels, which may serve as a strong prior for few-shot tasks. Combining self-supervised pretraining with meta-learning could reduce the need for large labeled meta-training sets and improve robustness.

7.2 Continual Few‑Shot Learning

Real-world systems must adapt to new tasks sequentially without forgetting previous ones. Continual few-shot learning merges the challenges of catastrophic forgetting and limited data. Future work may develop methods that maintain a growing set of prototypes or dynamically expand network capacity.

7.3 Cross‑Modal and Multi‑Task Few‑Shot Learning

Extending few-shot learning across modalities (e.g., image and text) or multiple tasks (e.g., classification and segmentation simultaneously) is a promising direction. Cross-modal few-shot learning could enable a model to recognize an object from a verbal description and a few images, while multi-task learning could share representations across related tasks for improved data efficiency.