Model / application / data / tools
LLM security testing across the complete AI application stack
LLM security is an application-security discipline with a probabilistic component. A strong assessment separates model behavior from the controls around it, then measures how failures propagate through data, identity, tools, and business workflows.
- Use this when
- You need to assess an AI feature across identity, application code, model behavior, retrieval, memory, data, tools, cost, and operations.
- You leave with
- A layered assurance plan that separates root causes while still testing the complete path a real user or attacker can reach.
First action
Reviewed 26 Jul 2026
Start the work before reading further.
Draw the request path from identity and input through prompt construction, model, retrieval, tools, storage, and final response.
Separate the layers before testing them together
The model layer includes instruction following, refusal behavior, uncertainty, and output consistency. The application layer includes identity, authorization, session state, prompt construction, parsers, rate limits, and audit trails. Retrieval and tool layers introduce their own data provenance and permission boundaries.
Testing each layer independently makes failures diagnosable. End-to-end tests then reveal interaction problems that unit-level checks miss, such as a safe model response being transformed into an unsafe tool action by application code.
Build an evaluation matrix tied to real impact
Use cases, assets, actors, and capabilities should drive the test matrix. A public writing assistant, an internal knowledge bot, and an autonomous operations agent do not share the same risks. Prioritize cases where the system can expose sensitive data, make consequential decisions, or perform external actions.
- 01Confidentiality: information disclosure, retrieval isolation, and memory separation.
- 02Integrity: instruction manipulation, poisoned context, and unauthorized state changes.
- 03Availability: resource exhaustion, looping tools, and dependency failure behavior.
- 04Accountability: identity binding, traceability, evidence quality, and administrative controls.
Treat reliability as a continuing measurement
Model updates, system-prompt changes, new tools, and retrieval-corpus changes can alter behavior without a traditional code diff. Keep a versioned evaluation corpus with expected boundaries, repeat it after meaningful changes, and watch both failure rate and user-impacting overblocking.
Operational readiness also includes cost controls, rate limits, safe failure behavior, credential isolation, observability, and a rollback path. These controls determine whether a model failure becomes a contained test result or a production incident.
Close the brief
Check the work before you move on.
- 01Assess model, application, retrieval, identity, and tool layers separately.
- 02Prioritize test cases according to reachable business impact.
- 03Version the evaluation corpus and rerun it after meaningful changes.
- 04Include cost, observability, failure behavior, and rollback in readiness reviews.
Evidence to keep
Each material risk has a reachable asset, a test at the correct layer, a measured result, an owner, and a regression case.
Next route