Luca Morelli
Software engineer and analyst
Team lead

AI application security: what it can see, what it can do, who oversees it.

An AI application reads documents, consults data and sometimes takes actions. More than on the model, the risks depend on three things the business decides: which data the system can reach, what it can do on its own and who oversees it.

Which data can it reach?

The system must not show a person data that this person could not consult directly.

This check is carried out by the application's code, not by the model: the instructions given to the model are part of the defence, but the decisive controls are enforced by the system.

A document, a web page or a message can contain hidden instructions, written to deceive the system (in the jargon, prompt injection). That is why everything the system reads must be treated as untrusted content, never as an order.

A concrete mistake: the question misread, silently

What can it do on its own?

If the system can take actions, such as sending a message, changing a piece of data or placing an order, for each one I define what it can do and with which permissions: the minimum necessary. That way a mistaken or hostile request does less damage.

  • Check who or what the action is directed at, before performing it.
  • Require a person's confirmation for important operations.
  • Set limits on attempts, duration and spending, with a point at which to stop.
  • Log actions, with attention to sensitive data, so that what happened can be reconstructed.

A system that can use more tools should not, for that reason, receive more authority.

Who oversees it, and what happens when it gets things wrong?

Alongside functional tests, I run hostile tests: requests built to deceive, attempts to access other people's data, content that tries to divert the system. Above all, I check that the limits hold when the model makes a mistake.

Every change to the model, data or instructions requires new tests. Who monitors the service, who receives reports and how an incident is handled must be decided before release. This is ongoing work, proportionate to the consequences of errors.

Verifying software developed with agents

In-house or from a provider: neither is secure by definition

A model run on your premises is not secure simply because it is in-house: it needs maintenance, updates and access control. An external service is not secure simply because it comes from a large provider. What counts is the actual configuration: where the model comes from and in which version, which components it uses, who has access to it and what path the data takes.

Data and constraints in the feasibility assessment

Why protections need reviewing over time

NIST's September 2026 assessment of GLM-5.3 shows advanced cyber-attack capabilities in a freely downloadable model, still below those of the best US models on aggregate tests. It signals that these capabilities are spreading, not how many attacks actually take place.

The implication I draw is to review periodically who you need to defend against: who can attack the system, with which tools, which data or functions they can reach and what damage they can cause. For a service reachable from the Internet, I therefore consider the long-standing practices even more important: fixing vulnerabilities, updating components, protecting credentials. The same AI tools can also help defenders, but their findings need verification.

The model comparison and its limits

Supporting sources

The cyber assessments describe the cited models and tests. OWASP references support the application risks; operational choices must be adapted to the actual system.

Content last updated

Assessing the risks of an AI application?

Detailed CV and references available on request.