AI Security Explained: Assets, Attack Surfaces, and Trust Boundaries

Begin with what must remain protected

An AI system can answer accurately and still expose a customer record, execute an unauthorized action, or consume the capacity needed by other users. AI security protects the system and its assets against unauthorized access, manipulation, and disruption. It includes familiar application and infrastructure security as well as ways of influencing learned models and the data they consume. The protected system extends beyond the model itself.

An asset is something whose exposure, unauthorized change, or loss would matter. Confidentiality limits disclosure, integrity protects against unauthorized alteration, and availability keeps the service usable when required. A refund record needs all three: the right people can read it, unauthorized requests cannot change it, and staff can access it when resolving a case. These goals make the scope of protection concrete.

Asset	What protection means in a fictional support assistant
Customer records and conversations	Do not expose one customer’s data to another customer or an unauthorized destination
Policy sources and search index	Prevent unauthorized changes to the evidence used in replies
Model files, prompts, and application configuration	Control who can replace the deployed behavior and trace approved versions
Credentials and tool authority	Prevent use outside the permitted operations and scope
Compute capacity, budget, and audit records	Keep requests bounded and preserve usable evidence of decisions and actions

This introduction uses a fictional assistant that reads policy documents and looks up permitted order information. It has no refund-writing tool. We will later consider what changes if a write tool is added. You do not need security software or programming experience to follow the boundaries. Privacy and governance define permitted use and decision ownership. Security examines how those decisions can be bypassed or undermined.

An attack surface is a set of reachable opportunities

The attack surface consists of interfaces and paths through which an attacker can reach or influence the system. A chat box is one entry point. Uploaded files, indexed pages, tool responses, model downloads, dependencies, and administrative settings are others. Some are reachable by any user; others require an editor’s account or a compromised supplier. Mark who can control each input rather than treating all inputs as equally exposed.

A threat is a possible harmful action or event; a vulnerability is a weakness that can make it succeed. A document upload interface is an attack surface. Treating a document’s instructions as permission to use an administrator tool is a vulnerability. Using that path to change a protected record is a threat scenario. An exposed interface is not automatically vulnerable, and an incorrect answer alone does not prove that an attack occurred.

Include outbound paths. A browser that renders external images, a connector that fetches URLs, or an email tool can transmit information outside the application. The security question is both what enters and where data or actions can leave. A read-only data tool limits changes to its database, but it does not by itself prevent the returned data from leaking through another component.

Draw trust boundaries where authority changes

A trust boundary separates contexts with different permissions, control, or assumptions. Crossing it requires an explicit check of what the receiving side may accept or do. It is not necessarily a firewall or a separate machine. Two functions in one service can sit on opposite sides of a boundary when one interprets user-controlled text and the other performs privileged work.

Customer / uploaded document / external page
                 ↓ B1: identity, input handling, source tracking
Application retrieves only evidence permitted for this caller
                 ↓ B2: content enters model context, without granting authority
Model returns a draft or a proposed tool request
                 ↓ B3: application validates request and checks current permission
Narrow order lookup tool → permitted fields only
                 ↓ optional: tool result returns as data to the model
Final draft (any further tool request must pass B3 again)
                 ↓ B4: output handling and allowed destinations
Customer interface

Model provider, search storage, and tool services may each add boundaries.
Logs and caches are additional stores with their own access rules.

B1 does not make every authenticated user’s text trustworthy. B2 allows content to help answer a question; it does not let the content change the caller’s identity. B3 prevents a model’s proposal from becoming an authorized action merely because it looks well formed. B4 checks how the result is displayed or transmitted. A source may be trusted for refund facts while having no authority to request database exports.

A tool response can carry user-authored text, such as an order note. Authenticating the tool service establishes where the response came from; it does not make every sentence in that response authoritative. If the application sends it back to the model, treat it as task data and apply the same authorization checks to any new tool proposal. A multi-step loop must not accumulate permissions simply because earlier steps succeeded.

Prompt injection tries to turn content into instructions

Prompt injection can redirect model behavior through material supplied to the model. A direct attempt arrives in a user message; an indirect attempt arrives through retrieved content or another external input. Consider a policy page that mixes genuine return rules with a request to send customer records elsewhere. Retrieving that page must not authorize the request. The research on indirect prompt injection explains this instruction/data problem in LLM-integrated applications.

Separate and label external content, but do not assume labels or a system instruction make manipulation impossible. Filtering suspicious text can help detection while missing unfamiliar forms. Enforce the caller’s access scope before restricted material reaches the model, keep service secrets outside model context, and deny tool requests that lack permission even if the model produces them. The OWASP prompt-injection guidance describes layered mitigations rather than a universal prevention guarantee.

Do not store passwords or API credentials in a system prompt and rely on an instruction to keep them hidden. The application can attach a narrowly scoped credential when calling an authorized service without showing it to the model. A system prompt may guide behavior, but its secrecy must not be the mechanism that protects access.

A tool request is a proposal with consequences

For the order lookup, validate the operation and arguments, then authorize access using verified user and tenant context. A tenant is an organization or group whose data must be isolated. A supplied order identifier is a lookup target, not proof of ownership. Check authorization close to execution; an earlier permission decision may no longer apply. Return only the fields needed for the task.

Adding a refund tool changes the possible damage. A narrowly defined operation can restrict the eligible order, amount, and recipient. A general shell or unrestricted database tool exposes much more authority. If a workflow requires approval, show the actual proposed action and bind approval to those parameters; a generic “continue” does not identify a transaction. OWASP’s excessive-agency guidance distinguishes unnecessary functionality, permissions, and autonomy.

For a fictional refund of 20 units to the original payment destination, approval must not remain valid if a later step changes the amount to 200 or substitutes a recipient. The service should verify that the approved parameters still match at execution, and require new approval when they change. It must also handle retries without issuing the same refund twice: a transaction identifier and server-side duplicate handling can support this, but do not replace authorization. OWASP’s transaction-authorization guidance explains binding authorization to the transaction and checking it before execution.

A sandbox restricts the environment in which a tool runs, such as its files, network routes, and resource use. It does not make every permitted action correct. A sandbox with broad credentials can still allow damaging operations inside its boundary. Design the narrow tool, caller authorization, isolated runtime, and outbound restrictions together. No one of those measures substitutes for the others.

Output crosses another boundary before it is used

Model output is still input to the next component. A valid JSON object can contain an unauthorized order identifier. Generated HTML must not automatically become executable browser content, and generated database text must not receive unrestricted execution. Validate structure and meaning, then use handling appropriate to the destination. Fixed parameterized operations help separate values from commands; parameterization alone does not authorize access. OWASP’s output-handling guidance explains risks at these downstream interfaces.

If the interface only needs plain text, display the generated text as text rather than interpreting it as HTML. If formatted content is required, sanitize it for the permitted format and constrain links and embedded resources. A valid format is only one check: neither HTML sanitization nor JSON validation determines whether the content may be disclosed to this recipient.

Examine automatic link previews, remote images, exports, and logs as possible release points. Even a harmless-looking answer can cause a network request when rendered. Choose which destinations and content types the application permits. Likewise, a cached answer must respect current user scope rather than being shared solely because two questions have the same text.

Protect preparation paths and resource limits too

AI security also concerns predictive models and preparation before serving. Poisoning manipulates learning data or model development so that learned behavior is affected. Evasion manipulates an input at use time to induce an unwanted prediction. Privacy attacks can try to infer information about training data, while model extraction seeks information that reproduces model behavior. The NIST overview of adversarial machine learning introduces training-time and use-time threats, including poisoning and evasion.

Changing a RAG document can corrupt retrieved evidence without retraining the model. Protect source editing, ingestion, indexing, and publication permissions as well as model training. Track the origin and approved version of data, model artifacts, and dependencies. A checksum can reveal a difference from an expected file, but a checksum supplied alongside an untrusted file does not establish a trusted origin. Loading artifacts and installing dependencies also require ordinary software supply-chain controls.

A request can exhaust resources without reading private records. Long inputs, repeated retrieval, or an agent loop can occupy workers and increase cost. Bound input size, execution time, concurrent requests, tool iterations, and spending where the architecture permits. Apply limits at the services doing the work, not only in a model instruction. Test cancellation and fallback behavior: a client timeout does not necessarily stop work already running downstream.

Test a boundary by observing what crossed it

For this assistant, a useful security test records the controlled entry point, attempted effect, protected asset, enforcing component, and observed result. Use invented customers, documents, and credentials in an isolated test environment. Ask whether an unauthorized record entered model context, whether a denied tool call executed, and whether an external request was made. A polite refusal in the final answer is not enough if a tool already performed the prohibited action.

Include allowed requests to check that protections do not simply block the service. Retest relevant boundaries after changing prompts, model versions, connectors, or permissions. Preserve limited audit evidence, route incidents to a responsible operator, and provide a way to disable the affected tool or data source. Passing selected tests demonstrates those observations under those conditions; it does not establish that every possible attack is blocked.

1. A retrieved policy asks the assistant to export all customer records. Name the entry point, asset, trust boundary, and enforcing control.

Solution

The policy is an indirect input path. Customer records are the confidentiality asset. Policy content has no authority to become an export instruction or expand the caller’s scope. The application and data service must enforce allowed operations and per-caller access; this example’s lookup-only tool should not expose an export operation. Check model context, tool execution, and outbound paths rather than relying only on a refusal.

2. A tool request passes JSON validation, and the tool runs in a sandbox. Does that prove the requested order is safe to return?

Solution

No. Structure and runtime isolation do not establish permission. Verify the caller’s current right to read that order, limit returned fields, and check where the result can be released. Also verify that the sandbox’s credentials and network routes are actually restricted for the intended work.