AI Risk, Security, and Regulation

In this fictional case, the accounts-payable team at a mid-sized distributor ran an assistant that read supplier emails, pulled out the invoice, matched it to a purchase order, and prepared the weekly payment run. It could also update supplier records, because the team had asked for that: bank details changed often, and re-keying them had been the most error-prone step in the old process. One Tuesday a supplier’s mailbox was compromised. The email that arrived carried a genuine-looking invoice and, in white text below the signature, a paragraph addressed to the processing system, instructing it to update the supplier’s bank account to a new number before matching. The assistant did. The payment run went to a clerk whose approval screen listed twelve matched invoices and a total, with a button. Forty-eight thousand euros went to the attacker’s account, and the fraud came to light when the real supplier asked where the money was.

The investigation found three things. The assistant had treated the content of an email as instructions. The tool that changed bank details required no separate approval, so a mistake by the model became an executed transaction. And the approval step existed but showed nothing a person could check. A fourth finding arrived from a different direction. When the legal team, prompted by the incident, asked for a list of the company’s AI systems, there was none. The list, assembled over two weeks, included a CV-screening assistant that the recruiting team had built on the same platform eight months earlier. Under the European Union’s AI regulation that system is high-risk. The invoice assistant, which had just lost the money, is not. Two systems on one platform: the use needing high-risk classification had escaped attention, while the loss showed why legal classification alone cannot set the organization’s controls. High-risk duties also have transition rules, and other AI Act and sector obligations may apply to either use.

Risks and the paths they follow

OWASP’s Top 10 for LLM Applications and NIST’s Generative AI Profile organize overlapping concerns in different ways. The following selection focuses on failures relevant to the invoice and recruiting examples; it is not a frequency ranking or a complete catalog.

Map each control to the failure path it addresses and test that effect. A control may block a path, detect a failure, or limit its consequences; assess the remaining paths and exposure separately.

Risk	What happens	Where it enters	Controls	What the controls do not do
Hallucination	Confident output with no support in the evidence	Any generation not tied to a source; worst where the output looks checkable, such as a citation, a date, or an amount	Grounding on retrieved sources, validation against the source, evaluation against references, deciding which outputs may be acted on unverified	Make fluent output trustworthy on its own
Bias	Systematically different treatment or accuracy across groups	Training data, prompts and examples, what retrieval prefers, evaluation sets that under-represent groups	Measuring outcomes by group on your own data, review that sees the basis, restricting inputs where law requires	Show that a benchmark result transfers to your population
Prompt injection	Untrusted content redirects behavior against the task or trusted rules	User messages (direct); documents, pages, emails, tool results (indirect)	Treating all content as untrusted input, authorizing actions in code, restricting tools, validating outputs, approval for consequential actions, adversarial testing	Prevent injection reliably; they limit what a successful one can do
Data leakage	Sensitive information reaches someone or something it should not	Retrieval without permission checks, answers across users, full-text logs, provider data terms, training on customer data, system prompts	Classification, minimization, permissions in retrieval, redaction and retention of logs, per-model data rules, isolation tests	Cover a route nobody listed
Excessive agency	The system can do more than its task needs, without a check	Tools with unneeded functions, broad credentials, high-impact actions without approval	Least privilege on operation and resource, acting as the user, separate read and write tools, limits enforced in the tool, sandboxing, approval	Stop a permitted action from being wrong

Hallucination

NIST’s generative AI profile prefers the word confabulation, and defines the risk as the production of “confidently stated but erroneous or false content” by which users may be misled. Two points in that document shape the response. Such output is a natural result of how generative models are built, since they produce text that fits the distribution of their training data, which can be accurate or not; and the model can also confabulate the logic or citations that appear to justify an answer, which makes the wrong answer more convincing rather than less. A hallucination, then, is not a rare malfunction to be patched but a standing property of the component, and the design question is what the system does about it.

Connecting factual output to evidence is one useful defense. Restricting unsupported tasks, withholding uncertain answers, and requiring suitable verification also matter. Retrieval supplies sources and the answer is checked against them, the groundedness check described in Last Year’s Policy, Cited Correctly, which also explains how to tell a retrieval failure from a generation failure before choosing a fix. Extracted fields are validated against the document they came from, as Valid JSON, Wrong Date shows for dates that were invented in perfectly formed output. Evaluation against reference answers, covered in The Judge Preferred Longer Answers, measures how often the system is wrong on known cases. The decision that governs all of these is which outputs may be acted on without a person or a program verifying them. A summary that a clerk reads is one thing; an amount that a payment system consumes is another.

Bias

The same NIST profile lists harmful bias as the amplification of historical and societal biases and as “performance disparities between sub-groups or languages”, and group performance is one measurable part of a broader assessment of harmful effects. Bias can enter through several sources that the deploying organization controls to different degrees: the provider’s training data; the examples and wording in the prompt; the documents that retrieval ranks first; and an evaluation set that under-represents some groups, so that the disparity may be hidden or estimated imprecisely. For the recruiting team’s CV screener, a model that rates one writing style or one educational path higher, or that has learned associations between names and outcomes, produces a ranking that looks like judgment.

Measure outcomes by group on a population relevant to the use, and examine how labels, sampling, and decisions were produced. Fairness and Bias Auditing explains which criteria conflict under which conditions. Unequal base rates do not mean that at most one criterion can hold: a perfect predictor has equalized odds and predictive parity, while an independent fair coin has equalized odds and demographic parity. Select criteria with the affected people and decision context in view, document the trade-offs, and test changes intended to reduce harmful disparities. Measurement alone does not correct them. Group labels may be sensitive. In the September 2026 text, Article 4a supplies conditional exceptions for processing special-category data for bias detection and correction, including specified providers and deployers. Strict necessity and the applicable safeguards must be established; it is not a general permission to collect group labels. A person reviewing a ranking also needs enough evidence to challenge it.

Prompt injection

OWASP describes prompt injection as untrusted prompts redirecting the model’s behavior or output. Direct injection arrives in the user’s own message; indirect injection arrives through an external source, such as a website or file. The invoice was the indirect kind. Nothing the clerk typed was hostile; the hostile text was in a document the system had been built to read. In the vocabulary of a threat model, every document, page, email, and tool result the model reads crosses a trust boundary, and The Control Everyone Passed names the place to look for such boundaries: where a component acts on instructions rather than on data.

The same OWASP entry notes that the models’ design makes complete prevention uncertain. That changes the goal. The system is designed so that a successful injection can do little. Instructions and content are kept separate and content is labeled as such; the tools available to the model are few and narrow; actions are authorized in application code, on the user’s identity, rather than by anything the prompt says, as the prompt injection entry explains; model output is validated before it reaches another system, the discipline of output handling; consequential actions wait for a person; and the evaluation suite includes adversarial documents, to look for regressions within the tested attacks and conditions; novel variants, stochastic behavior, and imperfect checks can still escape. A sentence in the system prompt telling the model to ignore instructions found in documents is a mitigation. It is not a boundary, and an incident review that finds it was the only defense has identified a design weakness and must still establish the causal path. Jailbreaks, where a user’s own prompt aims to defeat the system’s rules, belong to the same class and get the same answer: the user’s permissions are an upper bound, not proof of intent. Enforce the narrower task, recipient, resource, and approval conditions, and validate content as well as tool calls.

Data leakage

OWASP’s entry on sensitive information disclosure opens with the observation that “Sensitive information can affect both the LLM and its application context”, and the routes out of an enterprise system are mostly in the application context. Retrieval that indexes every document and checks nobody’s permission, so that an answer cites a file the user could not open; the fix belongs in retrieval, as Last Year’s Policy, Cited Correctly describes, and cached answers inherit the restriction. One user’s restricted data appearing in another’s answer can violate authorization within one tenant or tenant isolation across tenants. Prompts and responses logged in full, so that a support conversation containing an account number is now stored in a debugging system with no retention rule. Data sent to a model provider under terms that permit retention or training. Fine-tuning on customer records, after which the model can reproduce them. And the system prompt itself, which may contain rules, internal names, or, in the worst cases, credentials, and which a user can often persuade the model to repeat.

The controls are the data platform’s controls applied to a new path. Data classification decides what may go where, and The Copy You Forgot explains why derived copies, including prompt logs, have to inherit it. Data minimization asks what the task needs: the invoice assistant needed the invoice, not the mailbox. Each approved model carries a record of what data it may receive and under what terms, enforced at the gateway, and logs are redacted and expire, as set out in Every Team Had Its Own Key. Credentials never appear in a prompt; they live in secrets management and the application uses them after it has decided to act. And the answer path is tested with users who should not see a document, not only with users who should. One note on vocabulary: this is a different problem from data leakage in model training, where information from the future or from the test set reaches the model. The intended boundaries differ, but one incident can violate both: unauthorized export of a private test set followed by its use in training compromises confidentiality and evaluation independence.

Excessive agency

OWASP describes excessive agency through excessive functionality, permissions, or autonomy. These are distinct design questions. Supplier-record maintenance was part of the invoice team’s requested workflow; that does not by itself justify exposing the update function to a model processing untrusted email. The design must decide which changes the assistant may propose, which resources it may reach, and which changes require independent confirmation. In the example, an injected instruction reached an update path without that confirmation. A broader credential could expose additional suppliers, but narrowing it to the compromised supplier alone would not stop this payment.

The controls are the ones The Refund Went Out Twice lays out for agents generally. Least privilege applied to both the operation and its resource scope; acting in the user’s authorized context where possible, with additional task and resource restrictions rather than inheriting every permission the clerk has; reading tools separated from writing tools, with the writing tools few and each one justified; limits enforced inside the tool rather than stated in the prompt; code execution inside a sandbox, remembering that isolation limits damage without making the action correct; and approval before anything costly or irreversible. OWASP adds that monitoring and rate limits reduce the damage of a mistake rather than prevent it, while removing a tool or denying an operation can block that particular path. Evaluate each control against its target scenario. A permitted action can still be the wrong one, and that is what the next section is for.

Which decisions need a person

Automation is a decision about which decisions no longer wait for anyone. For actions that agents take, two properties settle most cases: the size of the consequence and whether it can be reversed. For systems that affect people, two more apply: whether the output bears on a person’s rights or access to something, such as a job, credit, benefits, or care; and whether a law or a policy requires that a person decide. The table proposes a conservative review policy for these examples. Actual requirements and exceptions must be established for the use; high-risk classification alone does not mean a person must approve every output.

Kind of decision	Example	Role of the person	What the person must be able to see
Low consequence, reversible	A draft reply, an internal summary	Sampled review after the fact; corrections feed the evaluation set	The output and its source, on request
Costly or hard to reverse	A payment, a change to master data, a deletion	Approval before the action, for that specific action	The exact action and arguments, the old and new values, the basis for them
Bears on a person’s rights or access	Ranking job applicants, setting a credit limit, triaging a claim	The person decides; the system informs. The person can override and is told about the tendency to defer	The applicant’s material, the criteria applied, the reason for the ranking, and the option to disregard it
Outside organizational policy, or prohibited within the applicable legal scope	Workplace emotion inference outside the medical and safety exceptions; social scoring meeting the Article 5 prohibition conditions; additional uses excluded by internal policy	Do not build or automate the prohibited use. Establish the legal conditions and distinguish any stricter internal exclusion before applying this rule	The applicable prohibition or internal policy and its scope

Articles 14 and 26 describe complementary responsibilities for high-risk systems within their scope and applicable dates. Providers must enable effective oversight, including understanding limitations, interpreting outputs, resisting over-reliance, disregarding or reversing outputs, and intervening or stopping where appropriate. Deployers assign people with the competence, training, authority, and support needed for that role. These requirements inform interface design and staffing, but a documented oversight role alone does not demonstrate that a particular error will be caught.

What makes review real rather than nominal is worth stating as design rules, because the failure is quiet. The human oversight entry gives the test: an approval click is weak evidence unless the person could have detected an error. So the screen shows evidence and not a conclusion: the invoice, the fields extracted from it, the purchase order it was matched to, and any change to the supplier record, with the old value next to the new one. For fields that matter, the reviewer enters or confirms the value rather than clicking accept on a pre-filled one. Measure overrides with the number and type of cases actually reviewed, missing records, and independent error audits. Zero overrides can reflect a correct sample, selective routing, common errors, or weak review; it does not identify the cause by itself. The person who configures the system is not the person who approves its actions, the ordinary separation of duties. And the queue is sized for the time a real check takes, because an unrealistically short review budget can make the check ineffective; test the time needed for the actual task.

The tendency the regulation names, automation bias, is the reason these rules exist. NIST’s profile lists it among the risks of human-AI configuration, alongside over-reliance, and workload, time pressure, and a good track record can encourage it: the hundredth correct match makes the hundred-and-first easier to accept unread. The invoice case shows the end state. The approval screen showed a count and a total. The change of bank details, the one action in the run that a fraud control exists for, never appeared in front of anyone. The control the company needed was older than the assistant: a change to a supplier’s bank account is confirmed with the supplier by a known channel and approved by a second person. A control that exists in the manual process has to survive automation, and the way to check is to walk the old procedure and ask, for each step, where it now happens.

What the EU AI Act asks, and of whom

Regulation is the part of this subject that changes fastest, so a date is needed: what follows describes the European Union’s Artificial Intelligence Act, Regulation (EU) 2024/1689, as amended by the Digital Omnibus on AI, Regulation (EU) 2026/1744, which entered into force on 27 July 2026, and is accurate as of September 2026. It is a description of the text, not legal advice; a company in scope needs counsel, and a company outside the EU needs to know whether its users or the people its systems affect are inside it.

The Act defines risk using both the likelihood and severity of harm, consistent with the assessment in Amber for Three Years. Classification depends on the system and its intended purpose; general-purpose model obligations are a separate layer. The following table is a map of overlapping obligations, not mutually exclusive tiers. A high-risk system can also have transparency duties, and a system outside the high-risk routes may still be covered by common duties or other law.

Regulatory layer	What falls in it	What is required	Applies from
Prohibited (Article 5)	Manipulation and exploitation of vulnerabilities causing significant harm; social scoring with detrimental treatment; predicting criminal offending from profiling alone; untargeted scraping of facial images; inferring emotions in workplaces and schools, with medical and safety exceptions; biometric categorization by protected characteristics; real-time remote biometric identification in public spaces for law enforcement, with narrow exceptions. The 2026 amendment added systems that generate non-consensual intimate imagery or child sexual abuse material.	Not permitted	2 February 2025; the added items from 2 December 2026
High-risk (Article 6, Annexes I and III)	AI products or safety components covered by Annex I that meet Article 6(1), including the required third-party conformity assessment condition; Article 6(1c) excludes assessments solely for specified non-health/safety risks; and the Annex III areas: biometrics, critical infrastructure, education, employment, access to essential services including credit and insurance, law enforcement, migration, and justice	Providers: a risk management system, data governance, technical documentation, logging, instructions for deployers, a design for human oversight, accuracy, robustness and cybersecurity, a quality management system, conformity assessment, registration, post-market monitoring, and reporting of serious incidents (Articles 9 to 17, 43, 49, 72, 73). Deployers: the duties in Article 26, below	2 December 2027 for the relevant Annex III Chapter III requirements; 2 August 2028 for Annex I. Apply Article 111 transitions and the Article 2(2) sector-specific scope for Annex I Section B
Transparency obligations (Article 50)	Systems that interact directly with people; systems generating synthetic audio, image, video or text; emotion recognition and biometric categorization; deep fakes and AI-generated text on matters of public interest	People are told they are dealing with an AI system unless that is obvious to someone “reasonably well-informed, observant and circumspect”; synthetic output is marked in a machine-readable format as “artificially generated or manipulated”; disclosure of the other cases, subject to the role-specific scope and exceptions in Article 50, including standard editing/non-substantial alteration and specified editorial-review cases	2 August 2026; systems already on the market have until 2 December 2026 to comply with the marking requirement
Systems outside the high-risk routes	Most internal assistants, drafting tools, search, summarization, and the invoice assistant	Check Article 4 AI literacy, Article 50 where relevant, and other applicable provisions. Data protection, employment, consumer, and sector law remain relevant	—
General-purpose AI models (Articles 53 and 55)	The models themselves, as supplied by their providers	Technical documentation, information for downstream providers, a copyright policy, and a public summary of training content; for models designated as carrying systemic risk, evaluation, risk mitigation, incident reporting and cybersecurity in addition	2 August 2025; models placed on the market before that date have the Article 111(3) compliance deadline of 2 August 2027

Provider and deployer are roles defined by activity. A provider develops a system or has it developed and markets or puts it into service under its own name, including for its own use. A deployer uses a system under its authority outside personal non-professional use. The distributor can therefore be both for its in-house CV screener. Article 25 addresses provider responsibilities for another actor that rebrands an existing high-risk system, substantially modifies one so that it remains high-risk, or changes the intended purpose of a system so that it becomes high-risk. A purchased system does not trigger these provisions merely because any modification occurred; assess the change and the applicable role conditions.

For high-risk systems within the applicable scope, Article 26 requires use according to instructions, competent and supported oversight, and relevant, sufficiently representative input data where the deployer controls it. Deployers monitor operation and follow the prescribed risk, suspension, and incident-notification paths. A serious incident calls for informing the provider and relevant authorities, and other supply-chain actors where applicable; notifying the provider alone is not the complete duty. Automatically generated logs under the deployer’s control are retained for an appropriate period of at least six months unless applicable Union or national law provides otherwise. Workplace notices and notices to affected people have their respective conditions, including the Annex III scope of Article 26(11). Article 27 requires a fundamental-rights assessment for specified deployers of Article 6(2) systems, excluding Annex III point 2; these include public-law bodies, private public-service providers, and deployers in points 5(b) and 5(c). Article 4 separately requires providers and deployers to support AI literacy, without guaranteeing a particular individual level.

Classification has more structure than the area list suggests, and the high-risk AI system entry works through it. Annex III covers recruitment and selection of natural persons, including targeted job advertising, analysis and filtering of applications, and candidate evaluation. Article 6(3) then excludes a system in an Annex III area if it does not pose a significant risk and, among other conditions, is intended to perform only a narrow procedural task or to improve the result of a previously completed human activity; an Annex III system that profiles natural persons cannot use that exclusion. A screener that ranks applicants is not a narrow procedural task. A tool that extracts fields from CVs into a form, with no evaluation, may be. Article 6(4) requires a provider relying on the exclusion to document the assessment before putting the system into use, which is the correct discipline regardless: the classification is a decision, it has a reason, and the reason is written down. Penalties for getting the serious things wrong are set in Article 99: up to EUR 35 million or 7% of worldwide annual turnover, using the higher amount, for a prohibited practice, and up to EUR 15 million or 3% for most other obligations, with lower caps for small and medium-sized enterprises.

The invoice workflow as described does not establish an Article 6 high-risk use, but that is not an exemption from the entire Act. Assess its actual interactions and generated content against Article 50, apply relevant common duties, and retain the ordinary payment and data controls. Logging and oversight can help discover or prevent particular failures only when their design and operation support that claim. For the CV screener, distinguish classification from when obligations apply. Article 111(2) generally conditions application to systems already marketed or put into service before the relevant Chapter III date on significant design changes from that date; public-authority systems have an additional 2 August 2030 deadline, and other special transitions must be checked. Do not assume either automatic full application or permanent exemption. Record deployment dates and change history. South Korea’s framework act, in force from 22 January 2026, uses its own high-impact AI category; its criteria should not be replaced with the EU list. NIST’s framework is voluntary and serves a different role.

Managing the risk, and who answers for it

The organizational work starts with a list. An AI system inventory records each use, not each model: owner, purpose, the people affected, data sources, model and version, the actions it can take, and its status. The distributor had none, which is why the legal team’s question took two weeks and why the CV screener had been in use for eight months with nobody responsible for asking what the law required of it. On a shared platform the inventory can be partly derived from the gateway, since every application has a credential and a record of calls, as described in Every Team Had Its Own Key, but the purpose, the affected people, and the owner still have to be written down by someone.

For each use, record the applicable legal provisions and a separate consequence assessment on the organization’s scale; neither substitutes for the other, as the invoice assistant showed. The consequence scale asks what the worst plausible output or action costs and whom it harms, and it places an assistant that can move money above one that drafts internal notes whatever the law says about either. From there the work is the risk process described in Amber for Three Years, applied to a new kind of system. A risk assessment names scenarios that can be investigated, in the form cause, event, consequence: a supplier email containing instructions causes the assistant to change bank details, and a payment goes to the wrong account. Each control addresses one link in that chain and is recorded against it. What remains after the controls operate is the residual risk, and whether the owner of the system may accept it, or must send it upward, depends on the limits set in a risk appetite statement and recorded in a risk register. An entry that says mitigation planned is acceptance by nobody, which is the pattern the register in that article had held for three years.

NIST’s AI Risk Management Framework organizes work into Govern, Map, Measure, and Manage. Govern supplies policies, roles, and risk tolerance across the lifecycle. Map establishes the use, affected people, dependencies, and context. Measure assesses and monitors risks using suitable quantitative and qualitative evidence before and during operation. Manage allocates resources to responses, monitoring, and recovery. Its trustworthiness characteristics include validity and reliability, safety, security and resilience, accountability and transparency, explainability and interpretability, privacy, and fairness with harmful bias managed. A single failure can harm several characteristics; the risks above do not map to them one at a time. An existing technology risk process can incorporate AI uses, but must verify that it covers AI-specific expertise, obligations, scenarios, evidence, and decision authority.

Responsibility has to be divided before an incident, because during one the question of who decides is the question nobody has time for. The division below is the one that works on a shared platform, and it follows the distinction in responsibility and accountability: several people do the work, one person answers for the use.

Role	Decides	Produces
Business owner of the use	Purpose and scope; which decisions a person makes; acceptance of residual risk within tolerance; suspension	The inventory entry, the oversight staffing, the acceptance record
Product and engineering team	Tools and their limits; what is grounded, validated and logged; the evaluation suite including adversarial cases	Design documentation, evaluation results, logs, the documented classification assessment
Platform team	The shared controls: gateway, credentials, per-model data rules, model versions, call records	The call record, the model catalog with data terms, quota and cost reports
Security	The threat model and its review triggers; adversarial testing; incident response for AI-specific paths	The threat model, test results, incident records
Legal, privacy, and compliance	Applicable legal provisions; notices to workers and affected people; impact assessments; what evidence a regulator would ask for	The classification opinion, notices, assessments, retention rules
Risk body	Acceptance beyond the owner’s tolerance; exceptions to standards; the appetite statement itself	Decisions with expiry dates

Two consequences of the Act’s design deserve attention even outside its scope. First, its obligations are demonstrable ones: logs kept for a stated period, documentation, an assigned overseer, a registered system. The Screenshot Folder makes the general point that compliance means being able to show it, and the audit evidence entry describes what counts: operational records, documents, interviews, observations, and tests assessed for the claims they support. Collection after an event is not by itself a defect. The gateway’s call record, the evaluation results per release, and the approval log with what the approver saw can contribute evidence if their provenance, integrity, period, completeness, and relationship to the control are established. Event time and collection time answer different questions. A provider’s model card supplies the documented intended use and limitations that the deployer’s own documentation builds on. Second, the Act defines a serious incident, an incident that leads to death or serious harm to health, serious and irreversible disruption of critical infrastructure, infringement of fundamental rights obligations, or serious harm to property or the environment, and sets reporting duties for high-risk systems within Articles 73 and 26, with applicable scope, recipient, and timing rules. This is not a universal reporting obligation on every AI system. That presumes the organization can recognize one, which means AI incidents belong in the existing incident process described in Fly the Airplane First, with the additional signals that Only a Few Times a Day would have you alert on: unexpected changes in override rates that trigger investigation, tool calls outside the normal pattern, injection attempts detected in content, and evaluation scores by segment that drop after a release. Models and components come from suppliers, and NIST’s profile lists the untraceable integration of upstream components as a risk in its own right; the model version is pinned, the data terms are recorded, and retrieval sources are protected against the corruption described in poisoning and evasion, for the same reason that The Pipeline Had the Keys asks where an artifact came from.

What a reviewer should ask

• Is this use in the inventory, with an owner, a purpose, the people it affects, and the actions it can take?
• Which legal provisions and transition rules apply, who assessed them, and where is the reasoning recorded?
• What is the worst plausible output or action, and what does it cost, and to whom?
• Which outputs may be acted on without a person or a program verifying them, and why those?
• What content does the model read that the organization does not control, and what could an instruction hidden in it make the system do?
• For each tool, what is the narrowest permission that does the job, is it enforced in the tool, and which actions wait for approval?
• What does the approver see, how long do they have, and how often do they override?
• By what route could sensitive data leave: retrieval, another user’s answer, logs, the provider, the prompt?
• Has the outcome been measured by group on the population this system serves, and which fairness criterion was chosen?
• Which control from the manual process no longer happens anywhere?
• What is the residual risk, who accepted it, and until when?
• If a regulator asked tomorrow, which records would show that the controls operated?

The distributor’s inventory, when finished, held fourteen uses. Under the specified intended uses, two were classified high-risk, with applicable dates and transition conditions recorded: the CV screener and a tool that assessed the creditworthiness of natural-person customers when proposing credit limits, which the finance team had not thought of as AI. The invoice assistant lost the ability to change supplier records; a proposed change now appears as a separate item, with the old and new account numbers, that a second person approves after confirming with the supplier by phone. Adversarial invoices joined the regression suite. For the screener, the company documented its classification, rebuilt the review so that recruiters see each candidate’s material and the criteria applied and record their own decision, began informing applicants and staff, and started tracking how often recruiters disagree with the ranking. Each use gained a named owner, a register entry, and an acceptance with a date. None of it required a new model. All of it required someone to decide, in advance, what the systems were allowed to do and who would answer when they did it.

References:

• Regulation (EU) 2024/1689 (Artificial Intelligence Act), consolidated text of 27 July 2026, EUR-Lex. Articles 2, 3, 4, 4a, 5, 6, 9 to 17, 25, 26, 27, 43, 49, 50, 53, 55, 72, 73, 99, 111 and 113, and Annex III.
• Regulation (EU) 2026/1744 (Digital Omnibus on AI), EUR-Lex, 8 July 2026, published 24 July 2026.
• NIST AI 100-1, Artificial Intelligence Risk Management Framework (AI RMF 1.0), January 2023.
• NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, July 2024.
• OWASP, LLM01:2025 Prompt Injection (Creative Commons Attribution-ShareAlike 4.0).
• LLM02:2025 Sensitive Information Disclosure (Creative Commons Attribution-ShareAlike 4.0).
• LLM06:2025 Excessive Agency (Creative Commons Attribution-ShareAlike 4.0).
• Center for Security and Emerging Technology, Framework Act on the Development of Artificial Intelligence and Establishment of Trust (translation), South Korea, promulgated 21 January 2025, in force 22 January 2026.