Leading an AI Incident: Containment to Resumption

In this fictional case, Harbor & Vale, a home-appliance service company, ran a chat assistant that answered warranty and order questions and could issue goodwill credits of up to \$100 per case. At 10:05 a.m. on a Friday a release gave the assistant a new capability: it could search the notes from closed service cases, so that it could say what a technician had already tried. At 4:40 p.m. a support agent forwarded a customer’s screenshot to the on-call channel. The assistant had given the customer another household’s name, street address, and phone number, along with that household’s dishwasher fault.

The head of AI transformation, who sponsored the assistant and was the executive on call, joined the channel at 4:55. Within a few minutes the engineers had reproduced the leak and found two more problems. The assistant was telling customers they had five years of warranty coverage, where policy gave one year, and it was approving credits above \$100. Switching the assistant off would stop all three problems and also send every chat to a support floor sized for fewer than four in ten of them. Switching off less would leave some harm running. The engineers could say how each switch worked. They could not say which cost the company should accept, who needed to be told by when, or what would have to be true before the assistant came back. Those were the executive’s decisions, and this article follows them from the first hour to the resumption.

What makes this an executive incident

The Organisation for Economic Co-operation and Development (OECD) defines an AI incident as an event in which the development, use, or malfunction of an AI system leads to actual harm: injury to health, disruption of critical infrastructure, violation of rights or of legal obligations that protect them, or harm to property, communities, or the environment. An event that could plausibly lead to such harm is an AI hazard. The distinction is practical. A hazard calls for a fix; an incident also calls for response to the harm that already occurred, including people who must be told and promises that may have to be honored. At Harbor & Vale the display of a household’s address is an incident for that household. The credit-limit flaw was a hazard until the first over-limit credit was issued.

AI incidents are quiet in a way that shapes the response. Every request in the Friday release completed with a normal status code and normal latency, so no availability alert fired. The harm was in the content and in the decisions. The signals that catch this and the engineering mechanics of rollback and recovery are covered in AI Platform Incident Response and Recovery. This article takes the other side of the same event: how an executive coordinates engineering, support operations, legal, communications, and finance while the engineers work.

NIST’s 2025 revision of its incident response guidance, SP 800-61 Rev. 3, takes the same position by treating incident response as part of cybersecurity risk management and not as a task handed to a specialist team. For an AI system that position has a concrete consequence: the response needs decisions that only the business can make, and they need owners before the incident. This article assumes the assignments described in Leading AI Transformation Across Business Functions are in place. At Harbor & Vale the director of support operations owns the customer-contact process, the head of AI transformation owns the assistant and its platform, and a named privacy counsel owns questions about personal data.

The first hour: who decides what

Command of the technical response belongs to a trained incident commander, and Fly the Airplane First describes the roles and the commander’s job in detail. The executive’s mistake to avoid is joining the technical thread and becoming a fourth person with a view on the cache. Her work is the set of decisions that carry business cost, and the first hour goes better when she knows in advance which ones are hers.

Some switches should need no executive at all. At Harbor & Vale the on-call engineer may disable any single tool the assistant uses, such as the credit tool, without approval, because a tool can be turned back on and the cost of a short outage is small. Turning the whole assistant off needs the director of support operations, because it moves the work to people and she is the one who must find them. The executive’s own decisions are the containment trade-off when the options differ in business cost, approval of anything said outside the company, whether to compensate or honor, and resumption. Decision rights written down before the incident are what make this a ten-minute conversation and not an argument in the channel.

One more habit pays off later. The commander or a delegate keeps a decision log with the time, the decision, who made it, the information available, and the alternatives rejected. Regulators, insurers, and the later review ask for exactly this record, and memory of a Friday evening does not supply it.

Containment is a choice among costs

Containment means stopping the harm from growing while the cause is still unknown; incident containment separates it from eradicating the cause and from recovery. The actions available to Harbor & Vale differ in what they stop, how fast they work, and how much work they push onto people. The log review found the following between the release and the 4:40 p.m. report, a window of 6 hours 35 minutes. All figures in this case are invented. 212 conversations displayed another household’s details, which is 32.2 per hour. 41 conversations promised five-year coverage. 63 credits exceeded the \$100 limit, which is 9.6 per hour, at an average of \$240.

The root of the first two problems was the same. The new retrieval step searched all closed-case notes with no check on which customer the note belonged to, and one note recorded a goodwill warranty extension granted to a single customer, which the assistant then presented as policy. The third problem came from the same release: the credit limit had been moved out of the credit tool and into the assistant’s instructions, where it was a request and not a rule. The support floor handles 180 of 600 weekday chats per hour in normal operation, since the assistant resolves 70 percent unaided. Forty-four agents at five chats per hour give a capacity of 220. Assume the contacts that depend on each capability do not overlap: 9 percent of chats involve a credit and 6 percent depend on case notes.

Option	What it stops	Minutes to take effect	Added contacts per hour for humans	Backlog growth per hour
Disable the credit tool	Over-limit credits only	2	54	14
Disable retrieval over case notes	The exposure and the false coverage claims	12	36	none (4 spare)
Both of the above	All three problems	12	90	50
Roll back the release	All three problems	40	0	none (40 spare)
Turn the assistant off	All three problems, and every other assistant function	5	420	380

Human load is 180 plus the added contacts; backlog growth is that load minus the capacity of 220. The minutes include the time to snapshot the index, prompt, and configuration for the investigation before anything is changed. The rollback is slow because it has to be verified, and that verification is what makes it a trustworthy end to the incident. A faster switch buys time for it. This is the usual relationship between a kill switch and a rollback.

At the observed rates, counted from the moment of decision, turning retrieval off takes 12 minutes and lets about 6.4 more conversations leak, and the credit tool is stopped after 2 minutes, with about 0.3 more over-limit credits. Relying on the rollback alone would let about 21.5 more conversations leak and 6.4 more over-limit credits through. Turning the whole assistant off at the five-minute mark would let 2.7 conversations and 0.8 credits through. Compared with disabling the two capabilities, switching everything off prevents about four more exposed conversations and one-half of a credit, and costs a backlog of 3,040 contacts after eight hours against 400. Eight hours at 380 contacts per hour is 3,040; at 50 per hour it is 400.

The arithmetic does not decide; it prices the choice. If the leaked content had been payment details or health information, four more exposures would justify almost any queue, and she would turn the assistant off. Here the exposed fields were a name, an address, a phone number, and an appliance fault, and the sponsor chose to disable the credit tool and retrieval, start the rollback in parallel, and keep both switches off until the rollback was verified. The choice was recorded with its reason and with the condition that would reverse it: if the log showed any leak after 5:20 p.m., the assistant would be turned off.

Keeping the business running

The director of support operations now carries the consequence. At 270 contacts per hour the floor needs 54 agents at five chats per hour, which is ten more than it has. Business continuity starts from what must survive: an agreed fallback with staffing, time, and error limits, so that the extra work is a decision and not an absorption. At Harbor & Vale the options are to pull ten trained agents from back-office work, which delays billing disputes, or to accept a queue. At a growth of 50 contacts per hour, the queue reaches 400 after eight hours, which the full floor would clear in about 1.8 hours.

Queue handling is also a safety matter. The company repairs gas and water appliances, so contacts that mention a gas smell or a leak go first, whatever arrived earlier. Customers waiting need an honest message about the delay and a callback option. A message that says the assistant is “temporarily unavailable” without saying how long the wait is tends to generate repeated contacts, which adds load to the problem the message was meant to relieve. The business also has to count what it cannot see: customers who gave up. The abandonment rate during the incident is a cost of the containment decision and belongs in the review.

Unsafe outputs and unsafe actions

The decision about what was already done is separate from containment, because containment stopped nothing that had already left. The 63 credits are done. Their excess over the limit is 63 × (\$240 − \$100) = \$8,820. Reversing them would mean debiting customers who were told by the company’s own assistant that they had received a credit, and the labor and goodwill cost of that is likely to exceed the money recovered. The director of support operations and finance recorded it as a loss, with the credits issued in the same window held for review in case any were not a goodwill case at all. Another question needs a sponsor’s answer: what else the assistant could do without a person. The credit tool was found in the first hour, and the commander also checked the assistant’s remaining tools for the same weakness, because excessive agency tends to appear in more than one place.

The false warranty statements raise a legal question that executives often answer by instinct. In 2024 a British Columbia tribunal considered a customer’s claim that Air Canada’s chatbot had wrongly told him he could apply retroactively for a bereavement fare. Air Canada argued that the chatbot was a separate entity responsible for its own statements. The tribunal rejected that, said the airline was responsible for all the information on its website whether it came from a static page or a chatbot, and ordered it to pay damages measured by the fare difference. A small-claims tribunal does not bind courts elsewhere, but the decision shows how the argument that an AI system is a separate speaker was received.

Harbor & Vale’s counsel advised that the 41 statements should be treated as the company’s own. The exposure is bounded: if every one of the 41 customers makes one repair claim at an average cost of \$310, the ceiling is 41 × \$310 = \$12,710, and that is an upper limit, since not every customer will claim. Against it stands the cost of disputing a promise the company’s assistant made, with a recording of the chat as evidence. The sponsor, finance, and counsel decided to honor the 41 as written, to contact each customer individually, and to treat any similar statement found later the same way. The decision belongs to the business and not to engineering, and it is cheaper to make on Saturday morning with a bounded count than after the first customer arrives with a screenshot.

Data exposure

Of the three problems, the exposure has the clock that someone else sets. Two counts matter and they are not the same. The review found 212 conversations that displayed another household’s details, involving the records of 187 distinct households, because some records appeared more than once, and 205 distinct customers saw them. The 187 are the people whose data was exposed, and the 205 are the people who now hold it. They need different messages and different follow-up, and a single “number affected” hides the distinction.

Where the General Data Protection Regulation applies, Article 33 requires a controller to notify the supervisory authority without undue delay and, where feasible, within 72 hours of becoming aware of a personal data breach, unless the breach is unlikely to result in a risk to individuals. The notification describes the nature of the breach, the categories and approximate number of people and records, the likely consequences, and the measures taken or proposed. If all of it is not yet known, it may be provided in phases. Article 33 also requires processors to tell the controller without undue delay, which matters when a vendor hosts the index or the model. Article 34 requires communication to the individuals, without undue delay, when the breach is likely to result in a high risk to them. A disclosure to a person who should not have received it is a breach even though no attacker was involved.

The European Data Protection Board’s guidelines treat a controller as aware when it has a reasonable degree of certainty that an incident compromised personal data, and they allow a short period of investigation to establish that. Counsel decides when that point was reached. At Harbor & Vale the screenshot arrived on Friday at 4:40 p.m. and the leak was reproduced with test accounts at 6:15 p.m. A 72-hour clock from 6:15 p.m. ends on Monday at 6:15 p.m., and one from 4:40 p.m. ends at 4:40 p.m. that day. The plan assumed the earlier time, which meant that privacy counsel and the person who files with the regulator had to be reachable on Saturday and Sunday. Other jurisdictions define a reportable event and set their own clocks. Counsel maps customers by location, and the executive makes sure that mapping starts on Friday night.

In March 2023 a bug in an open-source library caused ChatGPT to show some users the titles of other users’ conversations, and for about 1.2 percent of ChatGPT Plus subscribers active in a nine-hour window, billing details such as name, billing address, and partial card information could be visible to another user. OpenAI took the service offline, fixed the bug, and emailed the affected subscribers. The scope in that account came from a mechanism, a time window, and a count, and that is the form Harbor & Vale’s notice also needs. Reconstructing the scope was the first task the privacy counsel set the engineers: which records were retrieved, for which conversations, between which times.

Three other tasks follow. First, preserve the evidence before changing anything: the index, the prompt, the configuration, and the conversation logs. Those logs now contain extra personal data, so access to them is restricted to the investigation. Second, establish where the notes traveled. They were sent to the model provider as context, so the provider’s retention terms decide whether that is another copy to account for. Third, decide what to ask of the 205 recipients, who might reasonably be asked to delete a screenshot or message they received. The company asks, and does not demand.

Communicating with legal, customers, and the company

Counsel advises on obligations, and the executive decides everything above the minimum: whether to notify households that the law might not require to be notified, what to offer, and how to word it. Several audiences need different content, and one fact sheet maintained by the commander should feed all of them so the counts do not drift between messages. A number that changes between a regulator filing, a customer email, and a statement to staff invites the question of which one is true.

• Support agents learn first, because customers will ask them within hours. They get the plain facts, what to say, what not to speculate about, and where to send questions.
• The 187 households receive a notice, drafted with counsel, that says what was exposed in plain language, when, what the company has done, what they can do, and whom to contact. It does not describe the cause in technical terms.
• The 205 recipients receive a different message: they were shown information that was not theirs, the company asks them not to use or share it and to delete it, and it explains why.
• The 41 customers with false coverage statements are contacted individually and told that the company will honor what the assistant said.
• The regulator receives the notification Article 33 describes, with phased follow-up as the scope is established.
• Leadership, the board, the insurer, and the model vendor are told what each needs to make its own decisions or meet its own duties. Cyber and technology insurance policies often set notification conditions, so someone reads the policy on Saturday.

Two rules apply to all of them. Say what is known, what is not yet known, and when the next update will come, and keep the commitment. And do not say the problem is resolved until the resumption criteria are met: a message that declares the issue fixed on Saturday and is followed by a recurrence on Tuesday costs more than a message that said investigation continues. The company also does not attribute the failure to its model vendor in public statements. The customers dealt with Harbor & Vale, and the release that caused the harm was its own. The templates, update cadence, and the role of a communications lead are described in Fly the Airplane First.

A customer-service assistant like this one is generally outside the high-risk categories of the EU AI Act, whose serious-incident reporting duties apply to high-risk systems. A hiring screener or a credit-limit tool built on the same platform would be a different matter, and The Invoice Gave the Instructions works through how that classification is made. The first question for the privacy and legal team is therefore not what AI law says but what personal-data, consumer, contract, and sector obligations the event triggers.

The decision to resume

The pressure to resume starts when the first customer is on hold, so the criteria should be written while the queue is still manageable and by people who are not being asked to approve their own work. The decision splits into one decision per capability, because the capabilities have different risks and different readiness.

The rolled-back assistant returns first. It restores the 70 percent of chats that resolve without a person, and its prior version enforced the credit limit inside the tool. Its evidence is a regression suite run against the rolled-back build and a check that the production configuration matches what was tested. The credit tool returns with it only after a boundary test shows that the tool itself rejects a \$100.01 credit when the instructions say otherwise. The limit being a rule enforced where the action happens, and not a sentence the model reads, is the structural fix, and it is the same repair as in excessive agency: the control sits in the tool, so the model cannot talk its way past it.

Retrieval over case notes does not return on the same schedule. It needs a permission check that limits each query to records belonging to the customer in the conversation, a test that proves it, and a decision from the data owner about whether notes written for technicians should be searchable by a customer-facing assistant at all. The test is a cross-tenant leakage test: plant a unique marker in each household’s notes, act as every other household, and assert the marker never appears. It includes a positive check that each household can still reach its own notes, because a system that returns nothing also leaks nothing. Goodwill notes that record one-time exceptions need to be excluded or labeled so they cannot be read as policy.

Live traffic is a weak source of proof for rare failures, and the arithmetic shows why. If n independent trials produce no violations, the 95 percent upper bound on the violation rate is 1 − 0.05^(1/n), roughly 3/n: 0.30 percent after 1,000 trials and 0.10 percent after 3,000. Retrieval-dependent chats arrive at 36 per hour, so 3,000 live conversations take about 83 hours at full traffic. A stage that sends 10 percent of them takes ten times as long, about 833 hours. A rare leak can survive that long, and a bound of 0.10 percent would still allow one in a thousand. Structural controls and exhaustive test matrices carry the evidence, and live stages serve to detect gross failure and to confirm that production behaves like the test environment. Staged exposure still matters because it limits the harm of whatever the tests missed.

Each stage has an exit rule written in advance: the metric, the threshold, who watches it, and the automatic action if it is crossed. Any violation of a defined invariant, such as a conversation showing a record that does not belong to the customer, turns the switch off first and starts the review second. The switch is exercised before the stage starts, with its propagation time and its default measured, because an untested switch is an assumption.

The business owner, here the director of support operations, signs for the resumption, with the head of AI transformation, security, privacy counsel, and legal concurring in their areas. The engineers who built the change do not approve their own readiness. The sign-off sheet is short: each criterion, the evidence, and the person who accepted it. Human oversight applies to this decision too. A signature is weak evidence unless the signer saw the tests and could have said no.

How the weekend ended

At Harbor & Vale the credit tool was disabled at 5:02 p.m. and retrieval at 5:20 p.m. The rollback was verified at 6:05 p.m. The support floor ran with ten borrowed agents through the weekend and cleared its queue early Sunday. The privacy counsel filed an initial notification on Saturday afternoon, with scope to follow. The households were notified on Monday morning, and the 41 customers were contacted on Saturday. The rolled-back assistant and the credit tool returned on Sunday, after the regression suite and the \$100.01 boundary test passed. Retrieval over case notes stayed off for five weeks, until the permission filter, the leakage matrix, and the data owner’s decision on which note fields were searchable were all in place.

The review that followed was blameless in the sense described in blameless postmortem: it asked why a release could move the credit limit into a prompt, why the index had no per-customer filter, and why the release review and the canary did not look for either, and it did not ask who wrote the prompt. Its actions were assigned owners and dates. The decision log became its timeline. Nothing in it required a different model. What the weekend showed was that the company had engineers who could stop the assistant and nobody who had decided in advance what stopping it would cost, who could say so, and what had to be true before it started again.

References

• General Data Protection Regulation, Article 33: Notification of a personal data breach to the supervisory authority
• General Data Protection Regulation, Article 34: Communication of a personal data breach to the data subject
• NIST SP 800-61 Rev. 3, Incident Response Recommendations and Considerations for Cybersecurity Risk Management (April 2025)
• OECD.AI, Name it to tame it: Defining AI incidents and hazards
• Help Net Security, A bug revealed ChatGPT users’ chat history, personal and billing data (March 2023)
• McCarthy Tétrault, Moffatt v. Air Canada: A misrepresentation by an AI chatbot (discussion of 2024 BCCRT 149)