openmlsys-zh/v1/en_chapters/chapter_federated_learning/overview.md

Overview

With the rapid development of artificial intelligence, large-scale and high-quality data has become increasingly important for model performance and user experience. At the same time, data utilization has become a bottleneck constraining the further development of AI. Issues related to privacy, regulation, and engineering have prevented data sharing between devices, leading to the emergence of data silos. To address this challenge, Federated Learning (FL) was proposed. The concept of federated learning was first introduced in 2016. Under the requirements of user privacy protection, data security, and government regulations, federated learning enables effective machine learning modeling using data from multiple parties.

Definition

The core principle of federated learning is that data stays in place while the model moves. Clearly, centralizing data from all parties would fail to protect user privacy and would violate relevant laws and regulations. Federated learning allows the model to "move" across data holders, thereby enabling modeling without data leaving the local device. In federated learning, each party's data remains local, and a machine learning model is built by exchanging encrypted parameters or other information (on a central server).

Application Scenarios

In practical application scenarios, federated learning can be categorized based on the overlap of samples and features into horizontal federated learning (different samples, overlapping features), vertical federated learning (different features, overlapping samples), and federated transfer learning (neither samples nor features overlap).

Horizontal federated learning is suitable for scenarios where different participants possess the same features but different individuals. For example, in an advertising recommendation scenario, algorithm developers use data with the same features (click counts, dwell time, usage frequency, etc.) from different mobile phone users to build models. Since these feature data cannot leave the device, horizontal federated learning is used to jointly build models from multiple users' feature data.

Vertical federated learning is suitable for scenarios with substantial sample overlap but little feature overlap. For example, consider two different institutions: an insurance company and a hospital. Their user bases are likely to include most residents of the area, so the intersection of their users may be large. However, since the insurance company records users' financial behavior and credit ratings while the hospital holds users' disease and medication records, their feature intersection is small. Vertical federated learning aggregates these different features in an encrypted state to enhance model capability.

Federated transfer learning focuses on finding similarities between the source domain and the target domain. For example, consider two different institutions: a bank located in China and an e-commerce company located in the United States. Due to geographical limitations, the user base intersection of these two institutions is very small. Meanwhile, due to the different types of institutions, their data features also have only a small overlap. In this case, to conduct effective federated learning, transfer learning must be introduced. Federated transfer learning can address problems of small data scale on a single side and insufficient labeled samples, thereby improving model performance.

Deployment Scenarios

Federated learning is architecturally very similar to the parameter server approach (data center distributed learning), both employing a centralized server and decentralized clients to collaboratively build a machine learning model. Furthermore, depending on the deployment scenario, federated learning can be further divided into cross-silo and cross-device federated learning. Generally, cross-silo federated learning involves users at the enterprise or organizational level, while cross-device federated learning targets portable electronic devices and mobile devices. :numref:ch10-federated-learning-different-connection illustrates the differences and connections among the three approaches:

:width:800px 🏷️ch10-federated-learning-different-connection

Common Frameworks

As the demand for federated learning technology from users and developers continues to grow, the number of federated learning tools and frameworks has also been increasing. Below we introduce some mainstream federated learning frameworks.

TFF (TensorFlow Federated) is an open-source federated learning framework led by Google for machine learning and other computations on decentralized data. TFF was developed to facilitate open research and experimentation in federated learning. It trains shared global models among many participating clients who keep their training data locally. For example, federated learning has been used to train prediction models for mobile keyboards without uploading sensitive typing data to a server.

PaddleFL is an open-source federated learning framework based on PaddlePaddle, proposed by Baidu. Researchers can easily replicate and compare different federated learning algorithms using PaddleFL, and developers can readily deploy PaddleFL federated learning systems in large-scale distributed clusters. PaddleFL provides various federated learning strategies (horizontal federated learning, vertical federated learning) and their applications in computer vision, natural language processing, recommendation algorithms, and other domains. Additionally, PaddleFL offers applications for traditional machine learning training strategies, such as multi-task learning and transfer learning in federated learning environments. Leveraging PaddlePaddle's large-scale distributed training capabilities and Kubernetes' elastic scheduling of training tasks, PaddleFL can be easily deployed based on a full-stack open-source software.

FATE (Federated AI Technology Enabler), proposed by WeBank, is the world's first industrial-grade open-source federated learning framework that enables enterprises and institutions to collaborate on data while ensuring data security and privacy. The FATE project uses Secure Multi-Party Computation (MPC) and Homomorphic Encryption (HE) technologies to build underlying secure computation protocols, supporting secure computation for various types of machine learning, including logistic regression, tree-based algorithms, deep learning, and transfer learning. FATE was first open-sourced in February 2019, and the FATE community was established. Community members include major domestic cloud computing and financial services companies.

FedML is an open-source federated learning research and benchmark library led by the University of Southern California (USC), which facilitates the development of new federated learning algorithms and fair performance comparison. FedML supports three computing paradigms (distributed training, on-device training, and standalone simulation) for users to experiment in different system environments. FedML also facilitates diverse algorithmic research through flexible and general API design and reference baseline implementations. To enable fair comparison of various federated learning algorithms, FedML has set up comprehensive benchmark datasets, including non-Independent and Identically Distributed (IID) datasets.

PySyft is a secure and private deep learning Python library released by University College London (UCL), DeepMind, and OpenMined, encompassing federated learning, differential privacy, and multi-party learning. PySyft uses differential privacy and encrypted computation (MPC and HE) to decouple private data from model training.

Fedlearner is a vertical federated learning framework proposed by ByteDance that allows joint modeling on data distributed across institutions. Fedlearner comes with surrounding infrastructure for cluster management, job management, job monitoring, and network proxying. Fedlearner adopts a cloud-native deployment approach and stores data in HDFS. Fedlearner manages and launches tasks through Kubernetes. Both participating parties of each Fedlearner task need to simultaneously launch training tasks through Kubernetes, with a Master node uniformly managing multiple training tasks and Workers handling communication.

OpenFL is a Python framework for federated learning proposed by Intel. OpenFL aims to be a flexible, extensible, and easy-to-learn tool for data scientists.

Flower is an open-source federated learning system released by the University of Cambridge, primarily optimized for deploying federated learning algorithms on large-scale, heterogeneous devices.

MindSpore Fedrated is an open-source federated learning framework proposed by Huawei, supporting commercial deployment on tens of millions of stateless terminal devices, enabling full-scenario intelligent applications while keeping user data local. MindSpore Federated focuses on application scenarios of horizontal federated learning with large-scale participants, enabling users participating in federated learning to collaboratively build AI models without sharing local data. MindSpore Federated primarily addresses challenges in deploying federated learning in industrial scenarios, including privacy security, large-scale federated aggregation, semi-supervised federated learning, communication compression, and cross-platform deployment.