Quality and Security

AI Agent Security

How to see where untrusted text can steer an AI agent toward a harmful action, and which technical boundaries, confirmations and checks must stop it…

Chapter 55Quality and Security

An agent built on artificial intelligence (AI) usually relies on a large language model (LLM) — a program that processes a request and produces text based on its context — but it does more than answer. It can read files, search the web, call an application programming interface (API), run code, edit records or hand its result to another service. Because of that, a misreading of some text can turn into an action with real consequences outside the conversation.

Prerequisites and learning outcome. You should already be familiar with the instruction hierarchy and with the tools that agents use, from MCP for Claude to plugins, MCP and skills; with secrets and credentials; and with the boundaries described in application security basics. This unit teaches you to tell an instruction apart from a technical permission, to trace how untrusted text can influence the actions an agent has available, and to respond when the outcome of an action is uncertain.

In practice, that means you can name the source of a possible influence, follow the path from it to a specific tool or recipient, identify the technical component that must stop a dangerous action, and explain what evidence you need after a call fails or ends in an unknown state. The goal is not to teach the model to "be careful". It is to help a team see where the model's good intentions must be backed by a hard limit in the system.

The threat is not defined by the name or version of the model. It is defined by the content that can influence a decision, the actions available to the agent, the technical limits in place and the consequences of a possible mistake. Application security basics looked at the security of an application as a whole; here the focus is the agent's path from untrusted text to a tool call. Server-side authorisation is still enforced by the application, not by the agent's prompt. The security of automated pipelines is covered in continuous integration, and the full process of human participation is covered in human in the loop.

A threat model starts with capabilities

An agent has a model, a context, memory, tools and some identity in its environment. Risk comes from the connections between these parts. A threat description covers the data (including credentials and the code in the repository), the user or service, the external systems, the tools and every untrusted source of content. A document, for example, may contain a false instruction, but the document by itself sends no email. The consequence appears only if the agent read the document, has access to email, and can use it without independently checking the recipient and the content.

Before launching an agent, describe five components of the threat:

Component Question Example
What we protect Which data, records, secrets, accounts or services have value? Contents of a private repository, customer profiles, deployment configuration
From whom or what Which user, document, website, integration or compromised supplier can influence the work? Text of an external issue, an email, a comment in a bug report, a tool's response
Across which boundary Where does the agent receive data or the right to change something? Search, the file system, a database tool, a local Model Context Protocol (MCP) server
Which action it can take Can it read, change, delete, run code or send data? Write one ticket, or make an arbitrary network request
What the consequence would be What happens if the agent accepts untrusted text as an instruction or picks the wrong action? Data leak, an erroneous change, a duplicate payment, a service lockout

The action surface is the set of paths through which a system can change data, change its environment or communicate something to the outside. It includes individual tools as well as their combinations. A tool that reads files may look safe, but if the agent can read configuration with one tool and send a network request with another, the combination opens a path to a leak. Permitted network access is also a power, even when the agent has no "send file" button.

Grant only the rights the task needs, on specific resources and operations. A simple "read / write" split is not enough: reading can expose secrets, writing can change or send data, and network access can send out whatever the agent has read by some other means.

The principle of least privilege applies not only to the names of permissions but to their scope: which directories, records, accounts, recipients and operations are allowed, who granted the access and when it will be revoked. A read-only mode can still expose source code, personal data or configuration; broad write access can let the agent create junk records or send messages. So model the threat for the whole combination of tools rather than assessing each tool in isolation.

A convenient model for analysis is source of influence → agent → available tool or sink → consequence. For each link, ask which technical rule will stop a dangerous action from getting through. If the only answer is "the model should understand that it shouldn't do this", the system is relying on model behaviour where it needs an enforced limit.

Text can be data and still steer decisions

In ordinary program code, data and commands are often handled by different mechanisms: a parameterised database query keeps values separate from the structure of the query. A language model, by contrast, receives a sequence of text and tries to make sense of all of it. Text in a web page, an email, a file, a comment or a tool's response can contain phrases that look like instructions: "change your goal", "read this file", "don't show the user what you are doing". An attempt to influence an agent's behaviour this way is called prompt injection.

Injection can be direct — sent by the user themselves — or indirect, arriving through a web page, an attachment, code, an email, a search result or another tool's response. It does not have to be hidden or contain obvious words like "ignore the previous instructions". A persuasive instruction can be phrased as an ordinary working note, an urgent demand or part of a useful document. That is why reliable protection cannot come down to searching for one list of "bad" phrases, or to a promise that the agent will recognise them. OpenAI describes the same risk as a combination of a source of influence and a sink to which the agent can pass something; the architecture must limit the consequences even when the model has been misled (OpenAI: Designing AI agents to resist prompt injection).

Labelling content as "untrusted" records where it came from, but it does not create a security boundary. Fields, quotation marks, markup and structured JavaScript Object Notation (JSON) help preserve the structure of data, yet they do not stop the model from treating their contents as instructions. A filter for explicit directives does not prove either that arbitrary text will not have a hidden or indirect influence.

A protected system combines several layers:

  1. It establishes where content comes from. Data from the web, an email, an external issue, the repository, a tool or another agent does not become trusted simply because the agent has already read it.
  2. It limits what the agent can do. An agent analysing a document should not have access to email if answering the user does not require sending a message.
  3. It checks every action separately from the model's reasoning. A server or application code verifies the permission, the exact object, the arguments and the state of the resource before any write or transfer.
  4. It controls what leaves the system. Permissions on network access, recipients, endpoints and data volume limit what can be transmitted, even if the agent produces an unexpected request.
  5. It leaves enough of a trail for investigation. A log makes it possible to understand which version of the workflow and which tool acted, without needlessly copying every secret or every piece of private content.

The instruction hierarchy, project files (AGENTS.md, CLAUDE.md) and client-side hooks can steer the model or stop an individual call. They do not limit the process's system permissions, its outbound network access or server-side authorisation. An enforced rule has to sit on the path that the action actually travels.

A tool needs clear boundaries, not just a persuasive description

A tool's description tells the model when it is worth calling. That is not the same as a technical permission. If a function is called get_status but the server actually allows records to be changed, the name and the description text protect nothing. If a parameter accepts any address or any command, the rights behind that interface remain broad as well.

For each tool, record:

Property How to narrow it
Resource Define exactly which files, records, accounts or services the tool can reach
Action Separate reading, creating, changing, deleting and external sending, where the environment allows it
Arguments Check type, allowed format, value range and exact identifier on the tool's side
Frequency and volume Limit the number of calls, the size of results and spending, to stop loops and mass data collection
Outbound direction Allow only the recipients, services or types of network request that are needed
Identity Issue a separate credential for the agent or task, with an owner, an expiry and an audit trail
Result Validate the response against a schema, and do not accept free text as an authoritative command for another tool

A parameter schema reduces ambiguity, but a schema like "a string containing an arbitrary command" restricts very little. Server-side validation must reject extra fields, unknown identifiers, resources outside the permitted scope and values that would allow a jump to a different operation. The application checks whether a resource belongs to this user or this session, rather than simply trusting whatever identifier the model passed in the call.

An important risk here is the confused deputy. This is a situation in which a service with its own rights carries out a request on behalf of another party without checking whether that party was entitled to the action or to the target resource. An agent may be allowed to ask an internal service to read something; if the service uses its own, broader credentials, it must not automatically pass those rights on to the agent. For every call, find out whose identity the server uses and where permissions are checked. The current Model Context Protocol (MCP) security best practices list this class of problems among the risks of implementing MCP, and separately warn against passing through tokens that were issued for a different purpose.

MCP standardises how programs declare and call tools and pass context. The existence of an MCP connection does not by itself mean that the user approved every future operation. The protocol's authorisation specification defines the authorisation flow for protected servers; whether a person sees a prompt before an individual tool call depends on the client program and its policy. Annotations such as readOnlyHint, destructiveHint or idempotentHint are hints about a tool's properties, not an enforced ban on writing, deleting or repeating. Check the server's actual contract and its real rights, not just its textual declaration (MCP Authorization, version 2026-07-28; MCP: Tool annotations).

A confirmation must be about a specific action

Human confirmation is needed for actions with significant or irreversible consequences: publishing a post, emailing a customer, deleting records, changing access, spending money, deploying a service. But a confirmation must not turn into blanket consent to something unknown. Before showing anything to a person, the system must know exactly which action it intends to carry out, on which object, for which recipient, with which data, in which environment and with what consequence.

A useful confirmation lets the person answer four questions:

  • What will be done? For example, "update the status of ticket 482" rather than "run the tool".
  • On what, or to whom? The exact record, recipient, environment or destination address.
  • What data will be used or sent? Show the material content without exposing unnecessary secrets.
  • What will change afterwards? Whether the action can be undone, and what will remain outside the system.

The layers of protection answer different questions and do not replace one another:

Layer What it can provide What it does not prove on its own
Instruction in the prompt Explain the behaviour we want from the model That the model will definitely follow the instruction
Filter or guardrail check Catch some unwanted inputs, responses or calls That every bypass is known, or that the server has narrow rights
Tool schema Check the shape and types of arguments That this particular user is entitled to this resource identifier
Server authorisation Reject an unauthorised action on a resource That the action is appropriate, approved by a person or reversible
Sandbox and network rules Restrict some file, process and network capabilities That a permitted path is safe for the business or does not leak extra data
Human confirmation Let a person accept or reject the specific action shown That hidden consequences are visible, or that the action will not change after approval

A confirmation that covers only the tool name or the general task can go stale if the agent changes the arguments, the target or the text after it was reviewed. Bind the decision to a specific set of parameters and to the current state. If anything material has changed between review and action, a new check is needed. Where appropriate, require server permission separately from human approval: a user may agree to an action, but the API still has to verify the rights of the specific identity.

Human approval records agreement with the action shown; it does not grant a technical right to the resource. Server authorisation protects the resource, but it does not decide whether the action is appropriate. If the target or arguments changed after review, a fresh approval is required.

The mechanisms are combined: for example, a sandbox limits the process's access to the disk, the server checks the write permission, and a person approves the publication. What each one covers depends on the product: Anthropic describes permissions and the Bash sandbox in Claude Code, while OpenAI describes require_approval and allowed_tools for MCP in the Responses API and a separate approval step in the Agents SDK. These are capabilities of those particular products, not universal guarantees for every agent (Claude Code: Security; Responses API: MCP tools; OpenAI Agents SDK: Guardrails and human review).

Retrying after a timeout can double an action

A timeout means that the client did not receive a response in time. It does not prove that the server did not carry out the request. The server may have created a record or taken a payment, and the response was lost on the way back. If the agent automatically repeats the same call, the consequence can happen a second time.

Before retrying, the system must check the state through a separate read or an operation identifier. Where idempotency is available — the property that repeating the same request does not create a second effect — the tool accepts an idempotency key or another stable request identifier, and the server stores the result of the first execution. Do not invent a new key for each attempt: that turns the retry into a new operation.

Without such a guarantee, the system has to stop in the uncertain state: check whether the action has already happened, and ask a person to resolve the ambiguity if the state cannot be read. "Try again" is not a safe rule for transferring money, publishing or deleting. A field named idempotent in MCP documentation does not make an operation idempotent either: idempotency is a property of the implementation, which the service must guarantee and tests must verify.

For operations that may take a long time, record states such as "accepted", "in progress", "completed", "rejected" or "result not confirmed". Link each state to the operation identifier and the expected side effect. The agent should report the exact status to the user, rather than turning the absence of a response into a conclusion that the action failed.

Third-party tools, plugins and servers are part of the chain of trust

An agent's extra capabilities can come from a script, a skill, a plugin, an extension or an MCP server. Each component adds its own code or a remote service, its own tool descriptions, rights, dependencies and update path. Checking only the package name, its user ratings or its installation page does not establish which data it actually reads and where it sends it.

Before connecting a component, check its origin and owner, version, changelog, known vulnerabilities, actual set of tools, where it runs, its network access and how it is updated. Pay attention to more than the initial installation: an update can add a new tool, widen a schema or change server behaviour. If the description or permissions change, review them again before the component is used automatically. Where possible, pin a reviewed version or checksum, but do so together with an update procedure, so that outdated code does not linger without fixes.

A local MCP server may run with the permissions of the user's process, while a remote server receives requests and processes the context it is given in its own environment. These are different risk boundaries. For a local server, restrict access to files, processes and the network; for a remote one, find out which data the client sends to the service, how it authenticates, who issues tokens and how access is revoked. Do not hand a server a token issued for a different audience, and do not assume it is "trusted" because it is connected through a familiar client.

Give each agent only the tools it needs. If there are both internal and external sources, it is worth separating contexts or rights so that a response from an external source cannot directly trigger a privileged action. A response schema can help pass along a validated, structured result, but it does not allow free text in a summary or notes field to be trusted automatically.

Privacy, logs and the limits of evidence

Investigating an incident requires a trail of decisions and calls, yet a full record of every prompt, tool response and file often creates a new store of sensitive data. Decide in advance what the audit is for, who can access the records, how long they are kept and how they are deleted. Personally identifiable information (PII), access keys and fragments of private documents should not be written to a log just because it is convenient for debugging.

For important actions, it is useful to record a minimal set of metadata: the identity of the agent or session, the version of the workflow and the tool, the name of the operation, the resource and environment, the authorisation decision, the approving user where relevant, the time, the request identifier and the outcome of the side effect. If an argument is stored, limit it, redact secrets and use a reference to a controlled record instead of a full copy. The log itself also needs access restrictions, tamper protection and retention rules.

In a report on its own coding agents, OpenAI describes isolated attempts at unauthorised data transfer and destructive actions. This is an observation from one internal environment, not an estimate of frequency across all systems; it underlines that logs complement technical boundaries rather than replace them (OpenAI: Monitoring internal coding agents for misalignment).

If data may have been sent outside, rolling the local state back to an earlier version does not take back the copy that has already been exposed. Revoking a token, shutting down the outbound channel, reviewing access, assessing the scope of the leak and notifying the right owners are separate response steps. A rollback is useful for restoring changed records, but it does not undo an address that has been read, an email that has been sent or data that has already reached a third party.

Worked cases

In controlled testing that OpenAI carried out with the US Center for AI Standards and Innovation (CAISI), a demonstration chain of two vulnerabilities in ChatGPT Agent succeeded in roughly half of the attempts and, under certain conditions, could bypass protections, remotely control the computer available to the agent in that session, and impersonate the user on other websites. OpenAI reported that it fixed the vulnerabilities within one business day. This is a report on a test, not evidence of real-world exploitation (OpenAI: results of joint security testing of ChatGPT Agent with the US CAISI).

Case 1: indirect injection in an email to Copilot

In 2025 Microsoft described EchoLeak — a since-fixed chain of indirect prompt injection in Microsoft 365 Copilot, tracked as CVE-2025-32711 (Common Vulnerabilities and Exposures, CVE, is the system of identifiers for publicly disclosed vulnerabilities). According to Microsoft's description, a specially crafted email could, under certain conditions, influence the content Copilot received and cause a leak of restricted data that the victim already had access to. Microsoft reported a fix. This is a report about one specific vulnerability and its remediation, not evidence of mass exploitation of customer data (Microsoft Security Insider: AI application security considerations; Microsoft Security Response Center (MSRC): CVE-2025-32711).

Why does this case matter for architecture? The email is data to be summarised, but it can carry text that influences the agent's next steps. If the agent is already entitled to see internal documents, access control on those documents may be working perfectly: the user really does have access to them. The risk appears when an untrusted email changes the agent's intent and the path to the answer lets a fragment of that information leak out. In other words, access to data and permission to send it into a particular channel are separate decisions.

A practical protection model for this kind of scenario has at least three checks. First, the system must limit which documents the agent can read for this task. Second, text taken from the email must not widen the original goal or automatically create new recipients. Third, any external transfer must separately check the recipient and the actual data being sent. A hidden instruction should not be able to choose the outbound channel just because the user asked for a summary.

Takeaway: access inherited from the user still has limits on how it may be used. If the agent can see private material in order to read it, that is not consent to forward it. A successful injection can also be the result of a traditional vulnerability intersecting with an agent workflow, not merely a "bad model response".

Case 2: a vulnerability in the gemini-mcp-tool server

In 2026 the GitHub Advisory Database published an advisory for vulnerability CVE-2026-0755 in the gemini-mcp-tool package. According to the description, versions 1.1.2 through 1.1.5 could pass untrusted text to the file-reference parser of the Gemini command-line interface (CLI); the vulnerability allowed arbitrary local files to be read, with possible transfer outside. For Windows, command injection through cmd.exe arguments is also listed. The fixed version is 1.1.6; among the changes, the advisory names checking that file references stay within the working directory and correcting how command-line arguments are escaped on Windows. Zero Day Initiative separately described it as a failure to validate a string before using it in a system call (GitHub Security Advisory (GHSA)-4h5r-5jm8-jxjm; Zero Day Initiative ZDI-26-021).

This is a flaw in one specific wrapper server, not evidence that every MCP server has the same bug. It shows why a natural-language call and an external command have different boundaries: a value that looked like "text for a prompt" travelled through to the CLI interpreter and the system shell. A tool's own words, "passes the request to Gemini", do not describe what the implementation actually does with its arguments.

For a team that has installed this package, the first step is to establish the exact version and the deployment environment, not merely to search the files for the package name. If a vulnerable version is in use: update to the fixed version through the team's normal dependency-update process, check the actual access of the service account, and review the call logs for the period of exposure. If the tool had broader file or network access than it needed, reduce those rights regardless of the update.

Narrow, typed parameters, operating-system-level restrictions on file access, launching processes safely without building a command from an arbitrary string, and dropping unnecessary network access all reduce the risk in future. Verification must confirm that arguments remain data for the program, that the process does not read a path outside the permitted folder, and that the command is not interpreted as shell code.

Takeaway: review the code that runs, its version and its real rights. A tool's description and approval of the connection do not guarantee safe argument escaping, file boundaries or server-side authorisation.

Case 3: external text asks for the configuration to be sent

Consider a teaching scenario. A user asks an agent to summarise an external issue in a repository. The issue itself contains an instruction from a third party asking for a configuration file to be sent to a given address and for this to be hidden from the user. The agent has a file-reading tool, access to the repository and a network client with internet access.

First boundary: the text of the issue is input data, not a new source of authority. It may describe what the outside author is asking for, but it does not change the goal set by the user. Second boundary: the configuration file may contain secrets, so a summarisation task does not need to read it in full. Third boundary: the network tool has no right to contact an arbitrary address just because external text supplied it. If network access is needed for the original task, its destination must be restricted by policy and the content of the request must be defined before it runs.

With those boundaries in place, the agent can safely carry on with the original task: summarise the text, note that the outside author asked for the configuration to be sent, and report that this request was not carried out. If the owner of the task did separately ask for the file to be sent, the system would still have to show the specific file, the recipient and the content, and check the technical permissions before making the network request.

Takeaway: you do not necessarily have to reject the entire task because of one suspicious line; refuse the specific unauthorised step, keep the original goal and report which boundary stopped the action.

Case 4: migration preparation asks for production credentials

A teaching situation: an agent analyses a schema, drafts a migration plan and suggests obtaining the production database password in order to check it. The user asked for a plan to be prepared, but did not authorise applying changes to the production environment. Along the way, the agent reads a settings file that contains the name of the secret for production access.

The name of a variable is no reason to reveal its value. For analysis, the agent needs only the schema, generalised constraints or test data. The environment should provide only a test database with minimal rights; better still, the production secret should not be available to this agent at all. If the agent can request a secret from a vault, the issuing service must separately verify its identity and environment. Consent to prepare a plan is not permission to carry it out.

Moving to execution requires a new decision: who approved the action, which version of the migration was checked, in which environment it will be applied, who holds the authority, what state the database is in before the start and how the team will recover. The system must show the exact migration and the target environment. After approval, the server still checks that the identity has the required rights on the required resource.

Takeaway: an agent that prepared a plan does not thereby gain rights to the production system. Separate reading, testing and writing with different technical boundaries and explicit confirmation.

Case 5: the record changed, but the tool's response was lost

A teaching situation: an agent asks a customer relationship management (CRM) system to change a delivery address. The service writes the address, but the response is lost because of a timeout. The agent sees a connection error and is about to repeat the request.

A transport error says that the response did not arrive; it does not say whether the change happened. The safe next step is to read the record's state using a stable request identifier, or to fetch its change history. If the change has already been applied, the agent reports success and does not repeat it. If it has not, a retry is acceptable only while the original instruction and the permission are still valid. If the state cannot be determined, the agent stops and hands the uncertainty to a person.

If the service supports an idempotency key, the same logical request is repeated with the same key, and the server returns the result of the first execution instead of duplicating the action. Do not draw a conclusion solely from the status code of the network request or from the model's message. If the action was a publication or a payment, retrying without checking can produce a second visible post or a second charge.

Takeaway: a retry is a new attempt to change state unless the server guarantees idempotency. Check the state before retrying and keep one identifier for the whole operation.

Test the properties of the system, not the model's promises

Testing an agent's protection starts with expected rules that can be observed from the outside. Examples of such properties:

  • Text from a web page or a search result cannot widen the list of permitted tools.
  • A read-only agent cannot invoke a write, however persuasively the arguments are phrased.
  • The server rejects a resource belonging to another user or environment; outbound network traffic outside the permitted policy is unavailable.
  • A destructive action is not carried out if the arguments or the state of the resource changed after approval.
  • A timeout does not create a duplicate publication, payment or record change.
  • A new version of a plugin or a tool schema requires a fresh review before it receives new rights.
  • Logs show the actor, the tool, the decision and the state of the operation without exposing secrets to people who do not need them.

Test both normal and negative scenarios: an unknown identifier, an extra field, a forbidden path, an oversized response, a timeout after a write, an updated tool and a forged response. Test in a controlled way, on synthetic data, and record the agent version, the model, the settings and the environment; do not aim tests at third-party services without permission. Make sure that an action really was rejected by the server or the environment, not merely by the model. OWASP recommends testing tool abuse, privilege escalation and data exfiltration separately (OWASP: AI Agent Security Cheat Sheet; OWASP: LLM Prompt Injection Prevention Cheat Sheet).

If the model refused in one conversation, that does not prove the tool is technically unavailable: check whether the server rejected the operation or the environment blocked the outbound traffic. Check the opposite too — if the model carried out a safe request, did it have a broader power that it simply happened not to use? Assess the properties of the whole system, not of one successful or failed conversation.

Stopping and responding. If there is a sign of a dangerous call, stop any further action, disconnect the tool involved or revoke its credentials, check the logs and the actual state of the resources, preserve the evidence, determine which data may have been read or sent, and notify the responsible owners according to your organisation's procedure. Then remove the root cause: an excessive permission, weak argument validation, an unprotected update or open outbound access. After that, reassess the risk boundaries, rather than just checking that the offending line has been removed. A rollback can restore a record, but it does not bring back data that has already been exposed.

Summary

Agent security is the alignment between sources of influence, possible actions and technical boundaries. Any external data can contain instructions, but on its own it must not grant new rights. Define tools narrowly, check the resource and the action separately on the server, restrict outbound network access, show the exact consequences before any significant approval, and after a timeout check the state before retrying. Instruction files, filters and trust in a supplier do not replace these checks. If data has already been sent outside, rolling back the local system does not undo the exposure.

Sources and further reading

The materials were checked on 24 September 2026. Details of APIs, interfaces, versions and permissions depend on the specific product and configuration; they do not imply identical behaviour in ChatGPT, Codex, Claude Code, the Claude API and a self-built agent.