Security
Three OWASP LLM Risks We Can Test From Outside, and Three We Can't
By VirtuesTech Team ·
A product team adds an AI rewrite button. It takes the user's text, passes it to a model with some instructions, and returns the result. It ships in a sprint. In most API test plans, that endpoint gets the same treatment as any other: authentication checked, inputs fuzzed, done.
In one engagement we ran this year, that endpoint was the most serious finding.
What we found
The rewrite endpoint took a user-controlled parameter straight into the model prompt. Instructions placed in that parameter overrode the intended function, and the model followed them. The root cause was insufficient isolation between the system's instructions and the user's content.
On its own, that might sound contained. The same engagement also found two file-upload flaws: no allow-list of file types, and a null-byte trick that slipped disallowed files past the filter that did exist. All three were rated High. The client fixed them, adding prompt isolation on the AI endpoints and strict allow-list validation with content-signature checks on uploads, and a retest in June 2026 came back clean.
The finding is published in full, with severity, reproduction steps, and remediation guidance, on our report walkthrough page. The engagement itself is written up as a case study.
How that maps to the OWASP LLM Top 10
OWASP's 2025 list has ten categories. A test of a live AI feature, run through the same interfaces users and attackers reach, covers some of them directly and can't see others at all. It helps to be plain about which is which.
What an outside test covers well:
- LLM01 Prompt Injection. The finding above. Test every input that reaches the model, including the ones that arrive in files and retrieved documents.
- LLM07 System Prompt Leakage. If a user can override instructions, ask what else they can recover. Anything sensitive placed in a system prompt should be assumed readable.
- LLM05 Improper Output Handling. Follow the model's output. If it lands in a web page, a query, or another service without being treated as untrusted input, that is a separate flaw from the injection that produced it.
What it can't see:
- LLM03 Supply Chain and LLM04 Data and Model Poisoning live in how the model was sourced and trained. An endpoint test doesn't reach them, and a report that implies otherwise is overstating its coverage.
- LLM09 Misinformation is closer to a quality question than a security one. It belongs in functional testing of the AI feature, where outputs are checked against what the product is supposed to do.
The remaining categories (sensitive information disclosure, excessive agency, vector and embedding weaknesses, unbounded consumption) are testable where the application exposes them, which depends on how the feature is built. Our LLM security testing page maps all ten.
What we'd tell a team shipping an AI feature
- List every path into the model. Typed input is the obvious one. Uploaded files, retrieved documents, and tool results are the ones that get missed.
- Separate instructions from content. The fix in this engagement was isolation between the system prompt and the user's text. It is the control the rest depend on.
- Treat model output as untrusted. Whatever the model returns should be validated before anything else acts on it.
- Test the API around the model too. Two of the three High findings here were upload flaws, sitting next to the AI feature. Attackers chain what they find, which is why API security testing belongs in the same scope.
- Retest the fix against the original attack. A prompt change that blocks one phrasing often misses the next.
Category names are from the OWASP Top 10 for LLM Applications 2025.