Prompt injection
Text that tricks a language model into ignoring its instructions.
Instructions hidden in user input or in content the model reads (web pages, documents, emails) can make it reveal data, ignore its rules or take unintended actions. Controls include input and output filtering, least-privilege tool access and human approval for sensitive actions.
On the exam
A chatbot revealing its system prompt after a user’s crafted message is prompt injection.
Related terms
- Data poisoningCorrupting training data to manipulate a model.
- HallucinationConfident but false output from a generative AI model.
Test yourself with today’s question of the day.