Artificial Intelligence
Federated Learning — Decentralized ML
The privacy upgrade for Machine Learning
For many Machine Learning applications, tons of data is needed for it to work. The problem, however, is user data is sensitive and private.
Rising concerns of privacy and the call for data rights pose an issue for the traditional way of training and developing machine learning models, which require uses to hand over their sensitive data to a cloud server.
The solution? Federated learning.
Before we dive into today’s article, we have a new discord server where the bitgrit community discusses all things data science and AI, including our newly released BGR token! Join the server here!
What is Federated Learning?
Through a combination of distributed optimization, privacy research, and machine learning, federated learning was born.
A formal definition by Wikipedia:
Federated learning (also known as collaborative learning) is a machine learning technique that trains an algorithm across multiple decentralized edge devices or servers holding local data samples, without exchanging them.
The keywords here are decentralized and local data.
It was first introduced in a 2016 paper published by Google AI researchers: Communication-Efficient Learning of Deep Networks from Decentralized Data.
The main idea behind federated learning is bringing a centralized model to the decentralized devices, eliminating the need for acquiring user data.
Since user data doesn’t leave the device, this helps preserve data privacy and security, as only the model results are shared (more about that later).
This privacy upgrade for machine learning is revolutionary and opens up new opportunities for ML applications to handle sensitive data.
But before we dive into the use-cases and benefits of federated learning, let’s go through how it works with an example.
How it works
We’ll take the example of a next-word predictor for Google’s Gboard and go through the process of federated learning step-by-step.
First, Google built a base ML model in their cloud server, trained with public data.
Then, several user devices volunteer to train the model. They obtain the model by downloading it when connected to power and on a wi-fi network (as training the model is an intensive operation, and we wouldn’t want to drain the battery of users’ devices).
They provide relevant data to the model — keystrokes logs, feedback on prediction, etc. — for the model to learn and improve.
After it’s done training, any updates or improvement on the model is aggregated and encrypted and sent to the cloud, updating the base model with new information.
This cycle of downloading and updating happens on multiple devices and is repeated several times before reaching good accuracy. Only then is the model distributed to all other users for all types of use-cases.
An important note is that the training data still remains on the user’s device; only the training result is encrypted and sent to the cloud.
Here’s a breakdown of the process in an image.
Your phone personalizes the model locally, based on your usage ( A ). Many users’ updates are aggregated ( B ) to form a consensus change ( C ) to the shared model, after which the procedure is repeated. (source)
This collaborative way of training and developing machine learning models is powerful and has real-world applications.
Applications
Where data is siloed due to legal, economic reasons, etc., federated learning shines since it allows individual parties to train their models on a larger dataset.
1. Health Care
A perfect example is digital health. Data from medical institutions is kept siloed due to patient privacy and data governance and cannot be used without patient consent. With the traditional approach, machine learning models can only learn from the limited available data source and are biased towards the hospital’s instruments/demographics/practices.
With federated learning, the AI algorithms can gain more information from other hospitals, capturing more unbiased information such as genders, ages, demographics, etc. that can help the model make more generalizable predictions.
Read more → The future of digital health with federated learning
2. Self-driving cars
Self-driving cars can also be treated as individual actors, where the learning is brought to the cars instead of having data sent back to a central server.
Since driving in the real world is dangerous and often unpredictable, federated learning can accelerate the learning process, reducing the need to transfer high volumes of data. Ultimately, it has the potential to fast-track the process towards fully autonomous self-driving.
There are more applications of federated learning, primarily in the field of the Internet of Things (IoT) which echo the same message, which is about enabling IoT with ML while maintaining data privacy and reducing high communication and storage overhead.
Limits/Challenges
Federated learning is still a pretty novel idea, and some prevalent challenges stop it from reaching its full potential.
1. Non-iid data
Devices around the world are constantly generating data that is non-iid. In statistics literature, that means the data are not independent and non-identically distributed. The assumption of iid variables is central to many statistical methods and algorithms and can add complexity and cause problems to the model.
2. Device computational ability
Each device participating in the federated network differs in its capability on a software and hardware level (network connectivity, RAM, power, etc.). While most smartphones today are highly capable of computationally intensive tasks, like training a model, there is still a large portion of edge devices that aren’t capable and can slow down the device’s performance. There will be a tradeoff between maintaining the performance of the device and model accuracy.
3. Data labeling
Many supervised ML techniques require clear and consistent labels for the algorithm to perform. With data coming in from various devices, good data pipelines must be implemented to label the data automatically.
4. Data Leakage
Through reverse engineering, it’s still possible to identify and obtain the data from a specific user. However, privacy techniques such as differential privacy can strengthen the privacy of federated learning but at the cost of lower model accuracy.
For more challenges of federated learning, read here.
Frameworks
Interested in getting your hands dirty with federated learning?
Here are some frameworks for implementing federated learning.
Federated learning is a powerful idea in artificial intelligence. It allows for decentralized learning across multiple devices at low latency and power consumption while ensuring your data is private and secured.
It still faces challenges before it can be practically applied in the real world, but tons of research is being done today.
I hope this article kickstarted your interest in federated learning and gave you a glimpse into what it is and what it can achieve.
Below you can find more resources on federated learning.
Thanks for reading!
Further readings/resource
- Online comic on Federated Learning from Google
- Papers with code — Federated learning
- Federated Learning for Image Classification
- Federated Learning: Machine Learning on Decentralized Data (Google I/O’19)
References
- Federated Learning: Collaborative Machine Learning without Centralized Training Data
- What is federated learning?
Find us on these platforms 👇📱
