Security & Privacy
Model poisoning
Manipulating a model during training or updates so it behaves maliciously or incorrectly.
Example
Tampered model weights cause a released classifier to behave incorrectly.
Why it matters
It helps identify malicious changes introduced into the model-building process.
You'll hear it
“The behavior changed after that update, even on unchanged test cases.”
Can you control it?
Set by the team
The people building or running the AI choose this setup. An everyday user generally needs their help to change how this part works.
Common questions
- Does the attacker need to control the whole training process?
- Not always. Access to one influential component or update can be enough in some settings.
- Can a poisoned model still perform useful tasks?
- Yes. Harmful behavior may coexist with otherwise ordinary performance.
- Does passing a general benchmark rule it out?
- No. The targeted behavior may not appear in the benchmark's cases.