What is LLM security modelling?

It is the practice of mapping what an AI system can read, write and trigger, then constraining each so that a manipulated model causes limited damage. The unit of analysis is the system around the model, not the model itself.

What is prompt injection?

An attack where instructions are hidden in content the AI reads, such as an email, document or web page, causing it to act on the attacker's instruction instead of the user's. The OWASP GenAI Security Project ranks it the number one LLM application risk in its 2026 Top 10.

Can prompt injection be fully prevented?

No current vendor claims full prevention. Defences reduce it substantially, but sound practice is to design the surrounding system so that a fooled model cannot reach anything critical: least privilege, approval steps on consequential actions, and audit logging.

Does using enterprise AI tools remove the need for security modelling?

No. Enterprise tiers address data handling, training and retention, which matters, but they do not change what the model is connected to inside your business. Access scope, action limits and logging remain your responsibility.

Who should own LLM security in a mid-sized business?

A named owner with a written charter, supported by IT and legal. In our experience the absence of a named owner is the single most common gap, and no tooling compensates for it.

Steal our ideas

LLM Security Modelling: Assume the Model Will Be Fooled

LLM security modelling is the discipline of working out what an AI system can touch, what could go wrong when it is manipulated, and how to limit the damage. The current industry position, stated plainly in the OWASP GenAI Security Project's 2026 guidance, is that you should stop trying to build a model that cannot be fooled and instead build the system around it so that when the model is fooled, nothing important breaks.

That single sentence should reframe how any leadership team thinks about AI risk. The question is not whether your AI can be tricked. It can. The question is what it can reach when it is.

LLM security modelling (n.): the practice of mapping what an AI system can read, write and trigger, then constraining each of those so a manipulated model has a small blast radius. The unit of analysis is not the model. It is everything the model is connected to.

Why the model is the wrong place to put your trust

Language models follow instructions found in the text they read. That is what they are for, and it is also the core vulnerability: instructions can arrive hidden in an email a model summarises, a document it reads, or a web page it retrieves. The industry calls this prompt injection, and The OWASP GenAI Security Project has ranked it the number one risk to LLM applications in every edition of its Top 10, including the 2026 release.

No vendor has eliminated it. The defences are real and improving, but the honest engineering position is that a sufficiently motivated attacker can sometimes get an instruction through. Security therefore has to live in the system design, not in hoping the model behaves.

The four questions that model the risk

What can it read? Every data source the AI touches is potential leakage. If the model can read a folder, assume its contents can appear in an output.
What can it write or send? An AI that drafts is low risk. An AI that sends, posts, books or pays is a different category and needs approval steps.
Who can it act as? AI must inherit the permissions of the person using it, never more. A junior asking an AI must not see what a junior cannot see.
What is logged? If you cannot reconstruct what the AI read and produced for a given output, you cannot investigate an incident, and you cannot prove one did not happen.

Answer those four honestly for each AI tool in the business and you have a threat model a security team will recognise, written in language an executive can act on.

Blast-radius thinking in practice

The controls that follow from the four questions are unglamorous and effective: scoped access so each AI use case touches only the data it needs, allowlisted actions so the AI can only trigger approved operations, human approval on anything consequential or external, rate limits so a compromised process cannot run away, and audit logs on everything.

None of this requires new security theory. It is least privilege, the oldest idea in the field, applied to a new kind of actor. What is new is that the actor reads instructions from untrusted text, so the privileges have to be set as if it will occasionally follow the wrong ones.

The practical test we use: if this AI followed the worst instruction it could plausibly receive, what is the most expensive thing that happens? If the answer is unacceptable, the fix is narrowing what it can reach, not improving the prompt.

Where to start

Inventory every AI tool actually in use, including the unofficial ones. The unofficial ones are usually the risk.
Run the four questions against each. Most tools fall out as low risk quickly; attention concentrates on the few with broad read access or write ability.
Write the boundary down: a one-page usage policy and a data boundary list by data class, so staff know what is sanctioned without asking.
Log and review. A monthly look at what AI systems accessed is cheap insurance.

Questions people actually ask

What is LLM security modelling?: It is the practice of mapping what an AI system can read, write and trigger, then constraining each so that a manipulated model causes limited damage. The unit of analysis is the system around the model, not the model itself.
What is prompt injection?: An attack where instructions are hidden in content the AI reads, such as an email, document or web page, causing it to act on the attacker's instruction instead of the user's. The OWASP GenAI Security Project ranks it the number one LLM application risk in its 2026 Top 10.
Can prompt injection be fully prevented?: No current vendor claims full prevention. Defences reduce it substantially, but sound practice is to design the surrounding system so that a fooled model cannot reach anything critical: least privilege, approval steps on consequential actions, and audit logging.
Does using enterprise AI tools remove the need for security modelling?: No. Enterprise tiers address data handling, training and retention, which matters, but they do not change what the model is connected to inside your business. Access scope, action limits and logging remain your responsibility.
Who should own LLM security in a mid-sized business?: A named owner with a written charter, supported by IT and legal. In our experience the absence of a named owner is the single most common gap, and no tooling compensates for it.

Keep reading