Intro to Federated Learning

Join Federated Learning! In this two-part course series, you will use Flower, a popular open source framework, to build a federated learning system, and learn about federated fine-tuning of LLMs with private data in part two. Federated learning allows models to be trained across multiple devices or organizations without sharing data, improving privacy and security. Federated learning also has many practical uses, such as training next-word prediction models on mobile keyboards without transmitting sensitive keystrokes onto a central server. First, you’ll learn about the federated training process, how to tune and customize it, how to increase data privacy, and how to manage bandwidth usage in federated learning. Then, you’ll learn to apply federated learning to LLMs. You’ll explore challenges like data memorization and the computational resources required by LLMs, and explore techniques for efficiency and privacy enhancement, such as Parameter-Efficient Fine-Tuning (PEFT) and Differential Privacy (DP). This two-part course series is self-contained. If you already know what federated learning is, you can start directly with part two of the course. In detail, here’s what you’ll do in part one: 1. Learn how federated learning is used to train a variety of models, ranging from those for processing speech and vision all the way to the large language models, across distributed data while offering key data privacy options to users and organizations. 2. Learn how to train AI on distributed data by building, customizing, and tuning a federated learning project using Flower and PyTorch. 3. Gain intuition on how to think about Private Enhancing Technologies (PETs) in the context of federated learning, and work through an example using Differential Privacy, which protects individual data points from being traced back to their source. 4. Learn about two types of differential privacy – central and local – along with the dual approach of clipping and noising to protect private data. 5. Explore the bandwidth requirements for federated learning and how you can optimize it by reducing the update size and communication frequency.