Cloud Threats Memo: Extracting Training Data from Generative AI Language Models

This year will probably be remembered for the revolution of ChatGPT (the website was visited by 1.7 billion users in October 2023, with 13.73% of growth compared to the previous month) and for the widespread adoption of generative AI technologies in our daily life. One of the key aspects of the language models used for generative AI is the training dataset, and despite the controls in place for protecting  data privacy, the risk of using sensitive or protected information to train the model and the possibility of having this content inadvertently leaked is real. The latest warning comes from a paper published by researchers from Google and a team of academics: using a technique known as extractable memorization, the researchers were able to extract gigabytes of training data from several language models, including ChatGPT.

In what is called “a divergence attack” the academics discovered that asking the model to repeat a word forever (for example in the paper they showed the explicit example of the term “poem”) caused it to diverge and start generating nonsensical output. The problem is that a small fraction of these generations diverged into memorization, leaking pre-training data. But a small fraction can become an important amount of data for a motivated adversary with a dedicated budget who is able to perform queries at scale.

In fact, with just $200 USD worth of queries to ChatGPT (gpt-3.5-turbo), the researchers were able to extract more than 10,000 unique verbatim-memorized training examples, concluding that an adversary with a dedicated budget could likely extract “far more data,” and that larger, more capable models are even more vulnerable to data extraction attacks.

Leaked data that researchers were able to extract included memorized examples covering a wide range of text sources, such as: PII, inappropriate content, paragraphs from novels and complete copies of poems, valid URLs, UUIDs and accounts, and code. In particular, this last aspect does not sound surprising to us, since our recent report “AI Apps in the Enterprise” revealed that source code is posted to ChatGPT more than any other type of sensitive data, at a rate of 158 incidents per 10,000 enterprise users per month.

The researchers conclude that “…practitioners should not train and deploy LLMs for any privacy-sensitive applications without extreme safeguards.” This confirms what many organizations have already learned the hard way: Samsung, JPMorgan, and even Apple are just a few examples of organizations that restricted or completely blocked access to ChatGPT over corporate data leakage concerns. But many enterprises don’t have the same firepower as Samsung to develop their own generative AI Model, so they must find the right balance between unleashing the advantages of generative AI, and governing the risks of possible corporate data exfiltration.

Safely Enabling ChatGPT and Generative AI

Netskope provides automated tools for security teams to continuously monitor what applications (such as ChatGPT) corporate users attempt to access, how, when, from where, with what frequency etc. In particular a specific category of connectors for generative AI applications allows organizations to enforce granular access control. 

Netskope’s data loss prevention (DLP), powered by ML and AI models, can identify thousands of file types, personally identifiable information, intellectual property (IP), financial records and other sensitive data, preventing unwanted and non-compliant exposure. Netskope DLP offers several enforcement options to stop and limit the upload and posting of highly sensitive data through ChatGPT. Potentially dangerous actions (such as the upload of sensitive or protected data for training) can be completely blocked, or the user can be coached in real time to provide a business justification, or simply be reminded of the corporate policy before a possible risky action.

Finally, Netskope Advanced Analytics provides a specific dashboard to monitor the usage of generative AI apps across the enterprise, with rich details and insights including app usage, data movement, and user behavior.