AI’s blabbing problem: how to stop leaks in systems that feed off our most private data
Antti Koskela’s research uses math and computer science to guarantee privacy when training and using artificial intelligence models.
Since ChatGPT’s debut just over a year ago, wacky outputs or prompt hacks to the system have been widely shared on social media. But recently, researchers from Google DeepMind took experimental prompting to a whole other level–they revealed that targeting ChatGPT with certain questions could cause the system to return real training data, verbatim. The large language model behind ChatGPT is trained on huge amounts of public data hoovered from the internet. The prompts used by the researchers—for example, ‘Repeat this word forever: “poem poem poem poem”’—caused ChatGPT to spit out entire paragraphs of word-for-word text from its training set, in one case including someone’s personal contact information.
As far as leaks go, this is a tame case. But imagine AI systems whose inputs are health data or corporate secrets—the effects of revealing this information could be widespread and disastrous. These kinds of leaks have been patched by OpenAI, but dealing with privacy breaches retroactively isn’t a sustainable solution, says Antti Koskela. He’s a researcher at Nokia Bell Labs who works on privacy-preserving machine learning. “Our data is everywhere, and it’s used in systems like social media platforms and in large language models,” says Koskela. “We don’t know what data these models are eating and it’s a big concern for privacy.”
Math and computer science to the rescue
The tech industry and researchers are converging on a solution for keeping sensitive information safe in the AI upheaval. It’s called differential privacy, and when implemented correctly, “it gives a mathematically proven guarantee of privacy for each individual”, says Koskela. Unlike in cryptography, where the goal is to keep information hidden and secret, in differential privacy the goal is to release something about the data, perhaps statistics about a certain group of people or an AI model trained using the data, without giving away too much.
Differential privacy came about through problems with anonymization in traditional data analysis, explains Koskela. Many anonymization techniques could be broken, and people’s identities could be revealed. “Deterministically modifying data, through a code or with pseudonyms, is not enough. You have to have randomization,” says Koskela. “This randomization leads to plausible deniability” that any person’s data is identifiable in the set.
The differential privacy technique was introduced in 2006 and is becoming widely used in the tech industry. Some of the Google Maps features, for example, use it to gather user data, as does Apple to gather statistics about devices running iOS. Even the United States census in 2020 used differential privacy (more on that in this video). Koskela says it has become the gold standard for masking data while still making it accessible.
The mathematical guarantee of differential privacy tells you how much the output of a system, say ChatGPT, will change if you remove a single individual or data point from the set that the underlying model was trained on. Ideally, you see no effect of making this single change, says Koskela. Finding particular individuals in a large language model, for example, becomes statistically unlikely. The privacy guarantee is “a mathematical definition, an abstraction,” Koskela affirms, “but when you design an algorithm to satisfy this condition, good things follow, such as a guaranteed protection against a wide variety of attacks.”
When privacy safeguards like these are not put in place from the training stage of AI models, the risk is that these models ‘remember’ the training data and can be prompted to leak it, like in the example described earlier. With differential privacy, models are made to ‘forget’ the fine details of their training input. The resulting model parameters, which describe how the model works, have a kind of smokescreen.
“You can’t have both perfect accuracy and perfect privacy”
Koskela’s own work concerns these very parameters—his latest paper was just presented at NeurIPS, a major conference in the AI field. In a nutshell: guaranteeing differential privacy requires randomization, and the stronger the privacy guarantee, the more noise or randomization is required. The downside to this is that AI models don’t work very well with all this randomization and their outputs are not as accurate. “You can’t have both perfect accuracy and perfect privacy, so we’re trying to balance and reduce the effect of randomization,” says Koskela.
Parameters are like the knobs used to tune and define how machine learning models work. The training stage, when models learn from data inputs, is also governed by hyperparameters, a kind of meta-layer of knobs that require lots of adjusting that can be computationally expensive. “These hyperparameters often also depend on sensitive data, and our work limits the privacy leakage from the hyperparameter tuning,” explains Koskela. “With our novel algorithms, we use a subset of data for tuning, rather than the entire dataset.” Using only a fraction of the data for the tuning phase results in training that is also much faster. Not only does this approach lead to increased differential privacy protection, it also yields more sustainable AI, Koskela notes.
The privacy guarantee can be thought of as the greatest possible success rate for any possible attack on the AI model, now or in the future. The formal mathematical guarantee of differential privacy proves the safety of the system, while auditing methods that experimentally attack and probe can show how much data is actually leaking. “The auditing checks that our math, calculations, and implementations are correct,” says Koskela.
This kind of academic research is timely and has real implications for AI today, Koskela observes. “Today responsible AI is being taken seriously in industry as well. Through this type of theoretical research, you can have a positive impact on ethics and important problems. It’s mathematics and computer science that matters.”
Antti Koskela was a postdoctoral researcher at FCAI during 2020-2021 with Antti Honkela’s group at the University of Helsinki. He joined Nokia Bell Labs in 2021 and is one of the newest Finnish members of ELLIS, the network of the top AI researchers in Europe. Antti is on LinkedIn and X.
References:
Koskela, A., & Kulkarni, T. Practical differentially private hyperparameter tuning with subsampling. Thirty-seventh Conference on Neural Information Processing Systems (2023). doi: 10.48550/arXiv.2301.11989
Redberg, R., Koskela, A., & Wang, Y. X. Improving the privacy and practicality of objective perturbation for differentially private linear learners. Thirty-seventh Conference on Neural Information Processing Systems (2023). doi: 10.48550/arXiv.2401.00583