Definition
Few-Shot Learning represents the capability of machine learning models to learn new tasks from only a small number of labeled examples, typically ranging from one to ten examples per class.
Unlike traditional supervised learning requiring hundreds or thousands of examples per class to achieve reliable performance, few-shot learning systems rapidly adapt to new tasks with minimal data exposure.
The fundamental objective is to achieve high performance on novel tasks using drastically reduced amounts of task-specific training data, enabling practical machine learning applications where labeled data is scarce, expensive to obtain, or emerges dynamically.
This capability proves essential for real-world deployment where collecting large labeled datasets for every new task remains economically infeasible or time-prohibitive.1
Historical Development and Conceptual Foundation
Few-shot learning emerged from the observation that humans learn new visual categories from just one or two examples, while deep neural networks required thousands.
This inspiration motivated developing machine learning systems exhibiting similar sample efficiency.
Early approaches treated few-shot learning as a standard supervised learning problem with augmentation and regularization techniques, achieving modest improvements but remaining far below human performance.
The conceptual breakthrough came with meta-learning—learning-to-learn approaches where systems learn not specific tasks but rather learn the learning process itself.
Rather than training models to classify objects directly, meta-learning trains systems to learn classification algorithms from small datasets. This paradigm shift proved transformative, enabling systems to achieve far superior few-shot performance by accumulating knowledge about how to learn from limited data across diverse training tasks.
The rise of large pre-trained language models revealed an additional dimension: in-context learning. Large language models like GPT-3 demonstrated emergent few-shot abilities without explicit meta-learning training—providing task examples in prompts enabled models to solve new tasks without parameter updates.
This discovery demonstrated few-shot learning exists at the intersection of pretraining scale, prompt design, and learned world knowledge.
Meta-Learning Approaches
Meta-learning systems learn from collections of tasks (meta-tasks) how to rapidly adapt to new tasks. 2
Two primary meta-learning paradigms dominate:
Gradient-Based Meta-Learning (MAML) learns model initialization parameters enabling rapid task adaptation through few inner-loop gradient steps. Model-Agnostic Meta-Learning (MAML) optimizes initial parameters to minimize outer-loop loss after gradient steps on few examples.
During training, inner loops simulate task-specific gradient updates while outer loops optimize to minimize post-adaptation loss.
At test time, models apply inner-loop gradient steps to new task examples, achieving task-specific parameters quickly.
This approach proves model-agnostic—applicable to any differentiable architecture—and enables learning shared parameter spaces beneficial across tasks.3
Meta-SGD extends MAML by learning not only initialization parameters but also per-parameter learning rates, enabling different parameters to adapt at different speeds.
Task-specific parameters might require rapid updates while shared features need conservative adjustment, and learned learning rates capture these dynamics.
Metric-Based Meta-Learning (Prototypical Networks) learns embedding spaces where classification occurs via distance computations. Prototypical networks learn embeddings such that each class has a prototype (centroid) and test instances classify to the nearest prototype.
Unlike traditional nearest-neighbor classification using raw features, learned embeddings capture task-relevant similarities. Different tasks can have different decision boundaries—embeddings rearrange to optimize task-specific classification. This approach avoids inner-loop gradient computation, enabling fast test-time inference.
Extensions improve prototypical networks through dynamic memory mechanisms enabling multi-hop reasoning, multiple prototypes per class capturing intra-class variation, and diffusion-based prototype generation.
ProtoDiff uses task-guided diffusion to generate task-specific prototypes, gradually refining from random noise toward optimal prototypes conditioned on support examples.
Matching Networks learn end-to-end attention mechanisms comparing test and support examples. Rather than fixed prototypes, attention mechanisms determine which support examples matter for classifying each test instance, enabling instance-specific adaptation beyond class-level information.
Transfer Learning and Pre-training Approaches
An emerging paradigm combines large-scale pre-training with few-shot adaptation, sometimes outperforming explicit meta-learning. Models pre-trained on massive data develop rich feature representations that transfer well to new tasks with minimal fine-tuning.
Pre-Training Benefits: Pre-training enables models to learn general visual or linguistic features applicable across tasks. Fine-tuning only task-specific layers or using parameter-efficient adapters suffices for new tasks. This approach sometimes surpasses meta-learning, particularly on complex vision tasks, suggesting that general feature learning combined with task-specific adaptation proves more effective than explicit meta-learning of learning algorithms.
Meta-Baseline and Hybrid Approaches: combine pre-training and meta-learning. Models pre-train on base classes, then meta-train on novel task episodes. This staged approach benefits from both massive-scale pre-training and meta-learning optimization, achieving state-of-the-art few-shot performance.
In-Context Learning in Large Language Models: Large language models exhibit remarkable few-shot capabilities through in-context learning—providing task demonstrations in prompts enables solving new tasks without parameter updates.
Mechanism: Models encode task structure from examples and adapt internal computations accordingly. Mechanistic analysis reveals that in-context learning operates through attention patterns extracting relevant information from demonstrations, enabling rapid behavior adaptation.
Emergent Capability: Few-shot capabilities emerge from scale—small models show no in-context learning improvement while large models (100B+ parameters) demonstrate strong performance. This emergence suggests in-context learning requires sufficient capacity to extract and apply task patterns.
Comparison to Meta-Learning: In-context learning eliminates expensive meta-training—just use pre-trained models directly. However, it sometimes underperforms task-specific meta-training or fine-tuning, depending on prompt quality and model scale.
Zero-Shot Learning
Zero-shot learning extends the paradigm to recognizing classes with zero training examples.4 Rather than task examples, systems rely on class descriptions or semantic attributes.
Semantic Attributes: Hand-engineered attributes describing class properties enable knowledge transfer. A model learning relationships between visual features and attributes on seen classes can apply those relationships to unseen classes described by similar attributes.
Learned Embeddings: Word embeddings or CLIP representations provide semantic information for zero-shot classification without human annotation. Pre-trained language models enable interpreting textual descriptions naturally.
Generative ZSL: Some approaches synthesize visual features for unseen classes from semantic descriptions, enabling classification despite lack of training examples. Generative models learn mappings from semantic space to visual space, enabling synthesizing novel class examples.
Data Efficiency and Sample Complexity
Few-shot learning dramatically reduces data requirements compared to traditional learning:
Sample Complexity Reduction: Few-shot systems achieve with 10 examples what traditional systems require 1000+ examples to achieve, reducing labeled data costs by orders of magnitude.
Domain Transfer Benefits: Few-shot learning applies particularly well to domain adaptation—models pre-trained on source domains adapt to target domains with minimal target data. This enables deployment across diverse scenarios without substantial retraining.
Continual Learning Integration: Few-shot learning enables systems to continually learn new classes throughout deployment, adapting to non-stationary environments without forgetting previous knowledge.
Key Challenges
Overfitting to Few Examples: Small sample sizes enable overfitting where models memorize examples rather than generalizing. Regularization, data augmentation, and meta-learning objectives help constrain hypothesis spaces.
Domain Shift: Few-shot learning assumes test tasks resemble training meta-tasks. Performance degrades when test domains differ substantially (e.g., few-shot models trained on natural images struggle on medical images).
Task Distribution Shift: Techniques optimized for one type of few-shot task (classification) sometimes fail on others (regression, structured prediction).
Computational Efficiency: Meta-learning training requires solving many inner-loop optimization problems, increasing computational cost. Some approaches trade training cost for test efficiency or vice versa.
Applications Across Domains
Computer Vision: Few-shot image classification enables recognizing new object categories from minimal training images, essential for rapid deployment in dynamic environments.
Natural Language Processing: Few-shot Named Entity Recognition enables handling new entity types without extensive annotation. Prototypical networks and meta-learning approaches achieve competitive performance.
Speaker Verification: Few-shot speaker identification from short utterances enables voice-based authentication with minimal enrollment samples.
Medical Imaging: Few-shot approaches enable learning to diagnose rare diseases from limited patient scans, critical where data collection is expensive or unethical.
Fault Detection: Few-shot anomaly detection identifies new machine failure modes from minimal examples, enabling predictive maintenance without extensive historical data.
1 https://doi.org/10.48550/arXiv.1703.05175
2 https://doi.org/10.48550/arXiv.2403.03472
