AI application security: what it can see, what it can do, who oversees it.
An AI application reads documents, consults data and sometimes takes actions. More than on the model, the risks depend on three things the business decides: which data the system can reach, what it can do on its own and who oversees it.
Which data can it reach?
The system must not show a person data that this person could not consult directly.
This check is carried out by the application's code, not by the model: the instructions given to the model are part of the defence, but the decisive controls are enforced by the system.
A document, a web page or a message can contain hidden instructions, written to deceive the system (in the jargon, prompt injection). That is why everything the system reads must be treated as untrusted content, never as an order.
What can it do on its own?
If the system can take actions, such as sending a message, changing a piece of data or placing an order, for each one I define what it can do and with which permissions: the minimum necessary. That way a mistaken or hostile request does less damage.
- Check who or what the action is directed at, before performing it.
- Require a person's confirmation for important operations.
- Set limits on attempts, duration and spending, with a point at which to stop.
- Log actions, with attention to sensitive data, so that what happened can be reconstructed.
A system that can use more tools should not, for that reason, receive more authority.
Who oversees it, and what happens when it gets things wrong?
Alongside functional tests, I run hostile tests: requests built to deceive, attempts to access other people's data, content that tries to divert the system. Above all, I check that the limits hold when the model makes a mistake.
Every change to the model, data or instructions requires new tests. Who monitors the service, who receives reports and how an incident is handled must be decided before release. This is ongoing work, proportionate to the consequences of errors.
In-house or from a provider: neither is secure by definition
A model run on your premises is not secure simply because it is in-house: it needs maintenance, updates and access control. An external service is not secure simply because it comes from a large provider. What counts is the actual configuration: where the model comes from and in which version, which components it uses, who has access to it and what path the data takes.
Why protections need reviewing over time
NIST's September 2026 assessment of GLM-5.3 shows advanced cyber-attack capabilities in a freely downloadable model, still below those of the best US models on aggregate tests. It signals that these capabilities are spreading, not how many attacks actually take place.
The implication I draw is to review periodically who you need to defend against: who can attack the system, with which tools, which data or functions they can reach and what damage they can cause. For a service reachable from the Internet, I therefore consider the long-standing practices even more important: fixing vulnerabilities, updating components, protecting credentials. The same AI tools can also help defenders, but their findings need verification.
Supporting sources
The cyber assessments describe the cited models and tests. OWASP references support the application risks; operational choices must be adapted to the actual system.
- NIST CAISI — GLM-5.3 (opens in a new tab): cyber capabilities and limits of the frontier comparison.
- Anthropic — spread of cyber capabilities (opens in a new tab): exploit and safeguard assessments in test environments.
- OWASP — Prompt Injection (opens in a new tab): risks from hostile instructions in content processed by a model.
- OWASP — Excessive Agency (opens in a new tab): excessive actions, permissions and autonomy in systems with tools.