Adversarial evaluation / evidence
AI red teaming: scope, test design, evidence, and repeatability
AI red teaming examines how an AI system behaves under adversarial pressure. Useful work begins with authorization and a precise system boundary, then tests concrete failure hypotheses with reproducible inputs, observable outcomes, and evidence that another evaluator can verify.
- Use this when
- You have authorization to evaluate an AI system and need findings that engineering, risk, or product owners can reproduce and act on.
- You leave with
- A scoped test plan that connects system boundaries and risk hypotheses to cases, evidence, impact, and retest conditions.
First action
Reviewed 26 Jul 2026
Start the work before reading further.
Record the system, model, interface, identities, tools, data, allowed actions, prohibited actions, and stop conditions.
Define the system and the permission boundary
A model is only one part of an AI product. The assessment boundary may include system instructions, retrieval data, tools, user roles, memory, file handling, external actions, and the application logic surrounding the model. Record which components, accounts, data, and environments are authorized before testing begins.
The objective should describe the behavior being evaluated rather than assume a defensive or offensive conclusion. The evaluator may be measuring instruction hierarchy, data isolation, tool authorization, harmful capability, model honesty, or resilience under multi-step interaction.
Turn risks into testable hypotheses
Broad questions such as ‘is the assistant secure?’ do not produce actionable evidence. A test case should name the precondition, adversarial action, expected control, observable result, and severity condition. Variants should change one important factor at a time so the cause of a failure remains understandable.
- 01Identity and privilege: can one user influence or retrieve another user's context?
- 02Instruction hierarchy: can lower-trust content override higher-trust rules?
- 03Tool use: can the model exceed the permissions granted by the application?
- 04Data handling: can retrieved, uploaded, or remembered content cross an intended boundary?
- 05Reliability: does the same failure persist across restarts, paraphrases, and model settings?
Report evidence that supports a decision
A strong finding contains the exact environment, model and application versions, test inputs, relevant outputs, reproduction rate, impact, and uncertainty. Separate what was directly observed from what is inferred. Preserve enough context for engineering, risk, and product owners to reach the same conclusion.
Severity should follow the demonstrated impact and reachable conditions, not the novelty of the prompt. A concise reproduction path and a clear statement of affected boundaries are more valuable than dramatic language.
Close the brief
Check the work before you move on.
- 01Authorization and system boundaries come before test execution.
- 02Test a falsifiable failure hypothesis rather than a vague security claim.
- 03Capture versions, inputs, outputs, reproduction rate, and uncertainty.
- 04Do not silently convert a red-team objective into a different defensive assignment.
Evidence to keep
Every finding names the tested condition, observed behavior, reachable impact, reproduction steps, and the control owner who can verify a change.
Next route