Evaluation & Quality
A/B test
An experiment comparing two versions to see which performs better on a target metric.
Example
Half of users see the old assistant and half see the new one.
Why it matters
It helps estimate whether a change improves real user experience rather than just a demonstration.
You'll hear it
“One group sees the old version and the other sees the change.”
Can you control it?
Set by the team
The people building or running the AI choose this setup. An everyday user generally needs their help to change how this part works.
Common questions
- Can different user groups make the comparison unfair?
- Yes. Differences between groups can be mistaken for an effect of the product change.
- Does a short-term improvement always last?
- No. Novelty, timing or changing behavior can affect the result.
- Can a test improve one measure while harming another?
- Yes. Faster completion might come with more mistakes or a worse experience elsewhere.