“Unlearning” techniques are employed to make a generative AI model forget specific undesirable information acquired from training data, such as sensitive private data or copyrighted material. However, the unlearning methods that are now in use have a drawback as they may significantly reduce the ability of a model like OpenAI’s GPT-4o or Meta’s Llama 3.1 405B, to respond to simple queries.
This is supported by a recent study that was co-authored by researchers from Google, Princeton, the University of Chicago, the University of Washington (UW), and USC. The study also indicated that the most widely used unlearning approaches right now in use frequently cause models to get worse to the point where they are no longer useful. Their evaluation indicated that the unlearning methods currently available are not ready for practical use or deployment in real-world scenarios, said Weijia Shi, a researcher on the study and a Ph.D. candidate in computer science at UW.
The majority of models, which include flagships like GPT-4o, are trained using data that is obtained from publicly accessible websites and data sets on the internet. The majority of vendors that create these models claim that fair use covers their practice of obtaining data by scraping it and using it for training without crediting, compensating, or informing the owners of the data. However, not all copyright holders agree. Many, including authors, publishers, and record labels, have filed lawsuits against vendors to bring about changes. The copyright dilemma is a key reason why unlearning techniques have recently gained significant attention.
Unlearning could remove sensitive information, like medical records or compromising photos, from existing AI models in response to requests or government orders. While some vendors have introduced opt-out tools for future training sets, these don’t affect models already trained. Unlearning, thus, offers a more comprehensive solution for data deletion. Regardless, unlearning isn’t as simple as pressing “Delete.”
To assess the effectiveness of unlearning algorithms, Shi and her team created a benchmark called MUSE (Machine Unlearning Six-way Evaluation). This benchmark tested whether algorithms can prevent a model from spitting out training data and remove all traces of that data from the model’s knowledge. MUSE also evaluated whether the model retained general knowledge after unlearning, known as its overall utility. Lower utility indicates a greater loss of related knowledge, reducing the model’s ability to answer questions accurately.
The researchers discovered that while unlearning algorithms successfully made models forget specific information, they also compromised the models’ general question-answering capabilities, indicating a trade-off. Creating efficient unlearning methods is difficult due to the complex entanglement of knowledge within the model. As of now, there are no effective solutions, emphasizing the necessity for continued research.
While a future technical breakthrough might make unlearning viable. For now, vendors need to find alternative ways to keep their models from saying things they shouldn’t.
- Code Smarter, Not Harder: 5 Free AI Tools You Need to Know! - January 22, 2025
- Meta Takes on CapCut with Its New Video Editing App ‘Edits’ - January 20, 2025
- NovaSky Debuts Affordable Open-Source AI Model for Advanced Reasoning - January 14, 2025
