1.1 Definition and Core Concepts
Federated learning is a machine learning paradigm that enables collaborative model training across multiple decentralized clients—such as mobile devices, hospitals, or organizations—without requiring raw data to be transferred to a central server. Instead, each client trains a local model on its own data and shares only model updates (e.g., gradients or weights) with a coordinating server, which aggregates them to improve a shared global model. This approach addresses privacy concerns, reduces data transfer costs, and allows learning from distributed data sources while complying with data residency regulations.
1.2 Historical Development
1.2.1 Early Work on Decentralized Learning
The concept of decentralized machine learning predates the formalization of federated learning. Early research in the 1990s and 2000s explored distributed optimization algorithms, such as parallel stochastic gradient descent and consensus-based learning in multi-agent systems. These works laid the groundwork for training models across multiple nodes without centralizing data, though they primarily focused on performance rather than privacy.
1.2.2 Emergence of the Federated Learning Term
The term “federated learning” was popularized by Google in a 2016 paper describing a system for training models on mobile keyboard data. The key innovation was combining local training with secure aggregation to preserve user privacy. Subsequent research rapidly expanded the field, introducing formal privacy guarantees, communication-efficient protocols, and new application domains.
1.3 Relation to Privacy-Preserving Machine Learning
Federated learning is often categorized under the broader umbrella of privacy-preserving machine learning (PPML). While it inherently reduces data exposure by keeping raw data on clients, it is not inherently private. Model updates can leak sensitive information. Therefore, federated learning is frequently combined with other PPML techniques such as differential privacy, secure multi-party computation, and homomorphic encryption to strengthen privacy guarantees.
2.1 Client–Server Topology
2.1.1 Centralized Aggregation Server
The most common federated learning architecture uses a central aggregation server. This server maintains a global model and coordinates training rounds. It receives local updates from clients, performs an aggregation operation (e.g., weighted averaging), and broadcasts the updated global model back to the clients. The server does not have access to local raw data.
2.1.2 Clients and Local Training Loop
Each client holds a private dataset and runs local training, typically for a number of epochs or minibatch steps, on the current global model. The client computes an update (e.g., the difference between local and global model parameters) and sends that update to the server. After aggregation, the client receives an improved global model for the next round.
2.2 Communication Rounds
2.2.1 Global Aggregation Frequency
The global model is updated after each communication round, where a round consists of one server-to-client broadcast, local client training, and client-to-server update transmission. Aggregation frequency affects both convergence speed and communication cost. Higher frequency can lead to faster convergence but incurs greater overhead; lower frequency reduces communication but may cause client drift.
2.2.2 Bandwidth and Latency Considerations
Communication in federated learning is often constrained by client bandwidth and network latency. Transmitting large model updates (e.g., full gradients of a deep neural network) can be expensive. Compression techniques and asynchronous protocols are used to mitigate delays, especially in cross-device settings with heterogeneous network conditions.
2.3 Decentralized and Peer-to-Peer Variants
2.3.1 Gossip-Based Aggregation
In gossip-based federated learning, clients communicate with a random subset of peers rather than a central server. Each client averages its model with received models from neighbors, and the global consensus emerges through repeated interactions. This approach reduces reliance on a single point of failure and can improve resilience.
2.3.2 Fully Decentralized Topologies
Fully decentralized topologies eliminate the central server entirely. Each client acts as both a trainer and aggregator, updating its model based on information from a fixed or dynamic set of peers. Such architectures are useful in environments with strict data locality constraints or when no trusted server exists.
3.1 Horizontal Federated Learning
3.1.1 Overlapping Feature Spaces, Distinct Samples
Horizontal federated learning (HFL) applies when clients share the same feature space but have different sets of data samples. For example, several banks may each hold distinct customers but record the same types of financial attributes. HFL aggregates model updates across these clients, effectively increasing the sample size for training.
3.1.2 Use Case: Smartphone Keyboard Prediction
A classic example of HFL is training a next-word prediction model for smartphone keyboards. Millions of users each have their own typing data (distinct samples), but the input features (e.g., previous words) and output labels are the same across users. The global model learns common language patterns without uploading personal text.
3.2 Vertical Federated Learning
3.2.1 Overlapping Samples, Distinct Feature Spaces
Vertical federated learning (VFL) is used when clients have different attributes (features) on the same set of entities. For instance, a bank and an e-commerce company might both have data on the same customers but different columns (e.g., bank transaction history vs. browsing behavior). VFL allows them to collaboratively train a model without sharing raw feature values.
3.2.2 Use Case: Cross-Bank Credit Scoring
In credit scoring, a group of banks may each hold partial information about a common set of loan applicants. Using VFL, they can train a combined credit-risk model without exposing individual banks’ proprietary data fields. The model learns from the union of features across banks while keeping each bank’s dataset private.
3.3 Federated Transfer Learning
3.3.1 Non-Overlapping Data and Features
Federated transfer learning (FTL) addresses scenarios where clients have neither overlapping samples nor overlapping features. It leverages transfer learning techniques—such as representation learning across domains—to enable collaboration. A global model may learn a shared representation that can be fine-tuned for each client’s specific task.
3.3.2 Application in Healthcare with Different Institutions
For example, a hospital in one country and a research lab in another might have completely different patient populations and diagnostic tests. FTL allows them to train a model for a common disease prediction task by aligning feature embeddings, enabling knowledge transfer without direct data sharing.
4.1 Federated Averaging (FedAvg)
4.1.1 Weighted Averaging of Local Model Parameters
FedAvg is the foundational aggregation algorithm. In each round, the server selects a fraction of clients and sends them the current global model. Each selected client trains for several local epochs on its own data and returns the updated parameters. The server then computes a weighted average of these parameters, with weights proportional to each client’s dataset size.
4.1.2 Communication Efficiency and Convergence
By performing multiple local updates per round, FedAvg reduces the number of communication rounds needed for convergence. Its convergence properties are well-studied under convex and non-convex settings, though convergence can slow down in the presence of heterogeneous data. The algorithm remains the most widely used baseline in federated learning.
4.2 Advanced Aggregation Strategies
4.2.1 Personalization (e.g., FedPer, pFL)
Standard federated learning trains a single global model, but this may perform poorly on individual client data distributions. Personalization techniques adapt the global model to each client, for instance by splitting the model into shared base layers and client-specific head layers (FedPer) or using meta-learning frameworks (pFL). These methods improve accuracy on local tasks while maintaining some collaborative benefits.
4.2.2 Gradient Compression and Quantization
To reduce communication costs, clients can compress their updates before transmission. Common techniques include gradient quantization (e.g., converting 32-bit floats to 8-bit integers) and sparsification (sending only the largest-magnitude gradients). These methods introduce noise but can be combined with error accumulation to preserve convergence.
4.3 Robust and Secure Aggregation
4.3.1 Secure Multi-Party Computation (SMPC)
SMPC protocols allow multiple parties to compute a function over their private inputs without revealing those inputs. In federated learning, SMPC can be used to compute the aggregated model update from individual client updates, ensuring that even the aggregation server learns only the final aggregate, not individual updates.
4.3.2 Homomorphic Encryption Approaches
Homomorphic encryption enables computations on encrypted data. Clients can encrypt their model updates before sending them to the server, which performs aggregation on ciphertexts. The server learns only the decrypted aggregate, never individual updates. While computationally expensive, advances in partially homomorphic schemes have made this approach practical for some settings.
5.1 Privacy Threats in Federated Learning
5.1.1 Inference Attacks from Shared Gradients
Even though raw data is not shared, model updates can reveal private information. Attackers—malicious clients or servers—can infer sensitive attributes from gradients, such as the presence of a particular feature in a training sample. Gradient leakage attacks reconstruct entire input images in some cases.
5.1.2 Model Inversion and Membership Inference
Model inversion attacks attempt to reconstruct training data from the model itself. Membership inference attacks determine whether a specific data point was used in training. Both threats are heightened in federated settings because clients have only local access to their own data, but the server or other clients may attempt to extract information from the shared model.
5.2 Privacy-Preserving Techniques
5.2.1 Differential Privacy (DP)
Differential privacy adds calibrated noise to model updates, bounding the influence of any single data point. In federated learning, DP can be applied at the client level (local DP) or at the server level after aggregation (central DP). This provides formal guarantees but can reduce model accuracy.
5.2.2 Secure Aggregation Protocols
Secure aggregation protocols combine cryptographic techniques (e.g., secret sharing) to ensure that the server learns only the sum of client updates, not individual ones. This prevents gradient leakage from the server side. Protocols like Bonawitz et al. (2017) are designed to handle client dropout and provide computational efficiency.
5.3 Security Against Malicious Clients
5.3.1 Byzantine-Robust Aggregation
Malicious clients may send corrupted updates to poison the global model. Byzantine-robust aggregation methods, such as trimmed mean, Krum, or median-based rules, filter out outliers. These methods ensure that the aggregated model remains accurate even if a fraction of clients behave adversarially.
5.3.2 Anomaly Detection and Verification
Defenses include statistical monitoring of client updates (e.g., computing update norms or cosine similarities) and cryptographic proofs of correct behavior. Anomaly detection flags updates that deviate significantly from expected patterns, while verifiable computation can cryptographically enforce that clients executed the correct training algorithm.
6.1 Sources of Statistical Heterogeneity
6.1.1 Non-Independent and Identically Distributed (Non-IID) Data
In federated learning, data across clients is rarely independent and identically distributed. Each client’s local dataset may come from a different underlying distribution. For example, a keyboard app on different users shows varying language patterns. Non-IID data is the rule, not the exception.
6.1.2 Label Skew and Feature Skew
Two common forms of heterogeneity: label skew means the distribution of classes differs among clients (e.g., one hospital sees mostly cardiac cases, another sees mostly trauma). Feature skew means the same label may have different feature representations (e.g., images of the same object under different lighting conditions on different devices).
6.2 Challenges in Training Under Heterogeneity
6.2.1 Client Drift and Slow Convergence
When clients train on their own non-IID data, their local updates may drift in conflicting directions, causing the global model to converge slowly or to a poor optimum. This phenomenon, known as client drift, is a major source of performance degradation in federated learning.
6.2.2 Handling System Heterogeneity (Device Capabilities)
Clients may have varying computational power, memory, battery life, and network connectivity. Some devices may drop out during training, leading to incomplete updates. System heterogeneity requires protocols that are robust to stragglers and that can accommodate different local training budgets.
6.3 Mitigation Strategies
6.3.1 Meta-Learning Approaches
Meta-learning methods, such as Model-Agnostic Meta-Learning (MAML) applied to federated settings, aim to learn a global model that can be quickly adapted to each client’s local distribution. This reduces the harmful effects of heterogeneity by treating each client as a distinct task.
6.3.2 Adaptive Learning Rates and Clustering
Algorithms like FedProx introduce a proximal term to penalize large deviations from the global model. Clustering techniques group clients with similar data distributions and train separate models per cluster. Adaptive learning rates per client can also mitigate drift by adjusting the step size based on local data characteristics.
7.1 Reducing Communication Overhead
7.1.1 Gradient Sparsification and Quantization
Sparsification involves sending only a subset of gradient components (e.g., the top-k largest values), reducing transmitted data volume. Quantization reduces the bit width of each value. Both introduce noise, but error feedback mechanisms (e.g., memory of skipped gradients) help maintain convergence.
7.1.2 Local Steps and Minibatches
Increasing the number of local gradient steps per communication round decreases the frequency of updates. Similarly, larger local minibatches reduce variance but may increase computation time. Balancing local steps and communication rounds is key to minimizing overall wall-clock time.
7.2 Encoding and Compression Techniques
7.2.1 Top-k and Random Sampling
Top-k sparsification selects the k gradient entries with the largest absolute values. Random sampling chooses a random subset of coordinates. Both can be combined with quantization. Top-k tends to preserve model quality better, while random sampling can be simpler and more bandwidth-friendly.
7.2.2 Low-Rank Factorization
Model updates can be approximated by low-rank matrices, reducing the number of parameters to transmit. Techniques like matrix factorization or sketching are used to compress large weight matrices into smaller factors. This is particularly effective for deep neural network layers with high rank.
8.1 Healthcare and Biomedical Research
8.1.1 Disease Diagnosis Across Hospitals
Federated learning allows multiple hospitals to collaboratively train diagnostic models (e.g., for cancer detection from medical images) without sharing patient records. This improves model accuracy by pooling diverse patient populations while complying with strict privacy regulations like HIPAA and GDPR.
8.1.2 Drug Discovery with Distributed Genomic Data
Pharmaceutical companies and research institutions use federated learning to train models on genomic and proteomic data from different sources. By keeping sensitive genetic data on premises, they can develop predictive models for drug efficacy and toxicity without violating privacy or data governance policies.
8.2 Finance and Insurance
8.2.1 Fraud Detection Across Banks
Banks can jointly train a fraud detection model using horizontal federated learning, combining transaction patterns from multiple institutions. The resulting model detects novel fraud types that might be invisible from any single bank’s limited view while preserving customer privacy.
8.2.2 Credit Risk Modeling with Vertical FL
Vertical federated learning enables credit risk assessment by merging feature sets from different financial entities (e.g., bank transaction data and insurance payment history). This yields more accurate risk scores than any single provider could achieve alone.
8.3 Internet of Things (IoT) and Edge Computing
8.3.1 Smart Home Device Personalization
Smart speakers and other IoT devices can learn user preferences on-device using federated learning. For instance, voice recognition models improve over time without uploading audio recordings to the cloud, enhancing personalization while maintaining user privacy.
8.3.2 Predictive Maintenance in Industrial IoT
Manufacturing equipment sensors generate data at edge locations. Federated learning allows factories to train predictive maintenance models across multiple machines or plants, foreseeing failures based on aggregated operational patterns while keeping proprietary sensor data local.
8.4 Mobile and Consumer Devices
8.4.1 Language Model Training for Keyboards
Mobile keyboard apps are a flagship federated learning application. The on-device model learns word predictions and autocorrections from user typing history. Only anonymized, aggregated updates are sent to the server, and the global model is periodically downloaded to improve suggestions.
8.4.2 Recommendation Systems and Content Personalization
Social media and e-commerce platforms use federated learning to train recommendation models on users’ interactions (clicks, views, purchases). This enables personalized content without uploading users’ detailed activity logs, reducing privacy risks and data storage costs.
9.1 Scalability and Convergence Guarantees
Even with communication efficiency, federated learning faces scalability issues as the number of clients grows to millions. Theoretical convergence guarantees are often derived under simplifying assumptions (e.g., IID data, full participation) that do not hold in practice. Developing robust convergence theory and scalable orchestration remains an active area.
9.2 Fairness and Bias in Federated Data
Because data across clients is heterogeneous, the global model may exhibit bias against underrepresented groups. For example, if a medical model is trained mostly on data from urban hospitals, it may perform poorly on rural populations. Fairness metrics and bias mitigation strategies tailored to federated settings are still emerging.
9.3 Functional Encryption and Trust Assumptions
Many secure aggregation schemes rely on a trusted server or honest majority assumptions. In practice, the server may be compromised or collude with clients. Designing protocols that work under minimal trust assumptions (e.g., any number of malicious parties) without excessive computational overhead is an open challenge.
9.4 Interoperability and Standardization
As federated learning deployments proliferate, different platforms and frameworks use incompatible data formats, aggregation protocols, and privacy configurations. Lack of standardization hinders cross-organization collaboration and makes it difficult to combine federated learning components from different vendors.
10.1 Federated Learning with Foundation Models
Large foundation models (e.g., language models, vision transformers) are typically trained on massive centralized datasets. Adapting them to federated settings will require new techniques for efficient fine-tuning, prompt-based learning, and gradient compression at scale, enabling privacy-preserving use of powerful pre-trained models on distributed data.
10.2 On-Device Continual Learning
Federated learning naturally supports continual or lifelong learning: clients can keep training as new data arrives. Future systems will integrate proactive on-device learning with memory management and task adaptation, allowing models to evolve over time without forgetting previously learned patterns.
10.3 Integration with Blockchain and Federated Data Markets
Blockchain technology offers decentralized coordination and incentive mechanisms. Combining federated learning with smart contracts could enable transparent data markets where clients are rewarded for contributing high-quality model updates. This alignment of incentives may encourage broader participation in collaborative learning.
10.4 Ethical and Regulatory Considerations
As federated learning adoption grows, ethical questions around consent, data ownership, and algorithmic accountability will become critical. Regulatory frameworks (e.g., AI Act, HIPAA) will need to address federated systems explicitly. Future research will focus on building federated learning pipelines that are transparent, auditable, and compliant with evolving laws.