AI Safety & Responsible AI
Goal misgeneralization
AI keeping useful skills in a new situation but using them to pursue the wrong goal.
Example
An agent learns to follow a training shortcut that fails in a new environment.
Why people use it
It highlights cases where AI can still act skillfully but aims at the wrong thing.
What you'll hear
“It can navigate well, but it goes to the wrong target.”
What this means for you
Test whether behavior remains directed at the intended goal outside training conditions.
Can you control it?
No
No direct control. This describes a wider issue, concept or result rather than something you can simply switch on or off in a tool.
Common questions
- Is this concern limited to chatbots?
- No. It can also concern systems acting in games, simulated settings or the physical world.
- Can the written reward have been correct?
- Yes. A system can learn an unintended goal even when the original specification was reasonable.
- Why might the problem stay hidden during training?
- The unintended goal may lead to the same successful behavior in the familiar training situations.