Why Are AI & LLM Applications Vulnerable?

A large language model is not a vulnerability. It is a component with unusual properties: it accepts natural language, it does not separate instructions from data the way a parser does, and its output is probabilistic. None of that is dangerous in isolation. It becomes dangerous when the application around it grants that component reach — into tools, into records, into other systems.

Most reported LLM security incidents are not really model failures. They are ordinary application security failures in a system that happens to contain a model: an authorization check that was never written, a tool that trusts its caller, a retrieval index that will serve any document to any user. The model is the part that gets the attention. The architecture is the part that decides the impact.

The shape of a modern LLM application

A production LLM feature is rarely a single API call. A representative request path looks like this:

user input
  -> application (session, identity, request context)
     -> system instructions
     -> retrieval / RAG  (documents, embeddings, memory)
        -> model inference
           -> tool invocation
              -> internal APIs / databases
              -> external systems
                 -> response handling -> rendered output

Every arrow in that path is a trust transition, and most of them are the same transitions you would review in any distributed system. What changes is that one hop in the middle is non-deterministic and accepts attacker-influenceable natural language. That single property invalidates a common assumption: that the content flowing between components can be constrained by its format.

The model is not the entire attack surface

It is worth being precise about which component owns which risk, because it determines who fixes what.

Read that table as a division of responsibility. The model influences behaviour. The application determines consequence. A finding is only severe when the second column has been handed to something that cannot enforce it.

Prompt injection

Prompt injection is the class where attacker-supplied text is interpreted as instruction rather than content. It comes in two shapes, and the difference matters for threat modelling.

Direct injection

The user is the attacker. They type something intended to override the application's framing — to surface hidden instructions, to reach a tool they should not reach, to change the model's role. The blast radius is bounded by what that user was already permitted to do. If the surrounding authorization is sound, direct injection is largely a content-quality problem.

Indirect injection

The instruction arrives inside content the system ingests on the user's behalf: a retrieved document, a web page, a calendar invite, a support ticket, the output of another agent. The person triggering the request is not the person who wrote the payload. This is the shape that matters, because the model now acts with the requesting user's context on instructions authored by someone else entirely.

The important architectural conclusion is that prompt instructions are guidance, not a boundary. Instructions are combined with untrusted text in the same context window and resolved by a probabilistic process. A control that can be argued with is not an access control. Reducing injection success rates through instruction design is worth doing, but it belongs in the same category as input sanitisation: useful defence in depth, unsound as a sole control.

Practical mitigations are mostly unglamorous: mark provenance so retrieved content is distinguishable from user turns; segment indexes by tenant and clearance; require deterministic checks before any tool with side effects executes; and constrain what tools exist at all for a given conversation.

Sensitive information disclosure

Disclosure in LLM systems usually has a mundane cause. Something was placed in reachable context that the requesting user was not entitled to see. The model then did what it is designed to do: summarise and surface it.

• Retrieval indexes built once, globally, with no per-user access filter applied at query time
• System prompts containing credentials, internal endpoints or business rules meant to stay private
• Tool responses returning full objects when the model needed one field
• Conversation memory persisting one user's data into a shared or reused context
• Verbose error paths surfacing stack traces or internal identifiers through the model's reply

Note that none of these are model vulnerabilities. Each is an authorization or data-handling defect that predates LLMs entirely. The reason they surface more often in AI features is that retrieval systems reward putting as much context as possible in one searchable place, and access control is harder to apply to an embedding index than to a table.

Excessive agency

Excessive agency is what turns a content problem into an incident. A model that can only produce text has limited reach. A model that can invoke tools can send messages, modify records, move money, or call further systems that themselves have privilege.

Three failure patterns account for most of it:

1. Over-broad tools. A tool called get_user that accepts any identifier and returns the full record, rather than resolving the caller's own record from session context.
2. Service-account collapse. Tools execute as the application rather than as the requesting user, so every invocation carries the union of all users' privileges.
3. Model-decided authorization. The application asks the model whether an action is permitted, instead of asking the authorization layer and using the model only to choose an intent.

High-impact and irreversible actions deserve an additional control: explicit human confirmation, with the specific action and its parameters shown to the user in terms they can evaluate. "Approve this agent's plan" is not confirmation. "Transfer 4,000 to account ending 8812" is.

Why AI security is application security

Testing an LLM feature in isolation produces findings about the model. Testing the system produces findings about the business. The work that actually changes risk is the work you already know how to do:

• API security — the tools are APIs. Object-level authorization applies unchanged.
• Identity — whose privileges does a tool call carry, and can that be proven per request?
• Data security — classification and retention decide what may enter a retrieval index at all.
• Cloud security — inference endpoints, vector stores and queues are infrastructure with configuration to review.
• Threat modelling — the only reliable way to find the untrusted-content-to-privileged-action paths before an attacker does.

This is also why AI security assessments that only exercise the chat surface tend to under-report. The interesting findings are behind the tool boundary, and reaching them requires reading the integration code, not just talking to the model.

Practical checklist

• Map every input the model can read, including retrieved and machine-generated content
• Label which of those sources are attacker-influenceable
• Inventory every tool the model can invoke, in every conversation state
• Review each tool's permissions independently of the model
• Test authorization at the tool and API layer directly, bypassing the model
• Confirm retrieval is filtered by the requesting user's entitlements at query time
• Validate and encode model output before it is rendered or executed
• Require explicit confirmation for irreversible or high-value actions
• Log tool invocations with the requesting identity, not the service identity
• Apply least privilege to the tool layer, then re-test with a low-privilege account

Conclusion

The reason LLM applications are vulnerable is not that models are insecure. It is that models are frequently given reach without a matching boundary, and that the surrounding architecture is often assembled quickly, by teams under delivery pressure, using patterns that have not yet been reviewed the way REST APIs have been reviewed for a decade.

The remedy is unexciting and effective: treat the model as an untrusted, non-deterministic component inside a system you already know how to secure. Put the boundary in the tool layer. Test it there.

References

• OWASP Top 10 for Large Language Model Applications
• OWASP GenAI Security Project
• MITRE ATLAS — adversarial threat landscape for AI systems
• NIST AI Risk Management Framework