Framing an AI Application
Consider a hypothetical software company whose support team receives a few thousand tickets a week. Every ticket must reach the right queue (billing, account access, outages, how-to questions, or security) before anyone can start working on it. A person currently reads each ticket and routes it. Someone proposes that a language model do the routing, and the first demo looks good: ten tickets in, ten correct queues out.
Ten correct answers settle very little. They do not say whether the model is better than the keyword rules that a developer could write in an afternoon, or than the person doing the work today. They do not say what happens to the one ticket in fifty that is misrouted, how long a customer or an agent waits for the answer, or what number the team should see before it switches the system on. Those questions have to be answered before the model is integrated, because they determine what gets built, what gets measured, and when the team stops.
Answering them takes five steps for a task that an AI component will perform inside a larger workflow: describe the consumer of the output, draw the boundary of the task, build a non-AI baseline, set quality and latency targets, and price the errors so that acceptance becomes a checkable statement instead of an impression. The ticket-routing system is hypothetical, and all data in the examples is synthetic text generated by the code shown. The model’s outputs are produced by a labeled stand-in that simulates a classifier with chosen accuracy, so the numbers demonstrate the method and say nothing about how any real model performs. The examples assume general programming experience and the basic ideas of classification and evaluation from machine learning fundamentals.
Start from the consumer’s workflow
Every AI output has a consumer: a person or a program that reads the output and does something with it. The consumer, not the model, defines what a good output is. A summary read by an expert who will check the source is held to a different standard than a field extracted into a database that triggers a payment. Sculley and colleagues list undeclared consumers, systems that silently start depending on a model’s output, among the hidden costs of machine learning systems; naming the consumer up front removes that risk at its source.
For the ticket example, trace the path of one ticket through the workflow.
ticket arrives
|
v
triage: choose a queue <-- the step the AI component would perform
|
+--> billing queue ----+
+--> access queue -----+
+--> outage queue -----+--> an agent opens the ticket and works it
+--> how-to queue -----+
+--> security queue ---+The consumer of the triage step is the queue system, and behind it the agent who opens the ticket. Four questions about that consumer shape everything that follows.
- What decision does the output feed? Here it is a choice of queue. Each queue has a different staffing level and response target, so a ticket in the wrong queue waits for the wrong people.
- What does the consumer need in order to use the output? A queue system needs exactly one value from a closed list of five. It cannot use a paragraph of reasoning, and a free-text answer would force a second parsing step that can fail on its own.
- How quickly is the output needed? An agent who opens a ticket expects the suggested queue to be there almost at once, so the time budget is in seconds. A nightly batch that relabels old tickets has a budget of hours.
- What happens when the output is wrong, and who notices? A misrouted ticket may sit unseen until its response target has already been missed. Whether an error is caught depends on whether a person looks at the output before it takes effect.
The last question separates two designs of the same feature. If the model’s queue is shown to an agent as a suggestion that one click can override, a wrong answer costs a few seconds of agent attention and the agent is a safety net. If the model’s queue is applied automatically, a wrong answer has no safety net, and its cost is the full delay of the ticket in the wrong place. The model and the metric can be identical in both designs while the acceptable error rate differs by an order of magnitude. A person who approves outputs is also not automatically a safety net; the glossary entry on human oversight explains why review has to be designed so that the reviewer can actually detect errors.
The table summarizes how common consumers change the framing.
| Consumer | Typical output shape | What an error costs | What the framing must fix |
|---|---|---|---|
| A person reads the output as a suggestion | Short text or a ranked list | Seconds of attention; trust erodes if errors are frequent | Latency budget; how often the suggestion is accepted unchanged |
| A person approves the output before it takes effect | A proposed action with its evidence | Reviewer time; approval fatigue can let errors through | Review capacity; what the reviewer sees |
| A program parses the output | A schema-valid structure from a closed set | Silent corruption downstream if values are wrong but valid | Output contract; validation; cost of valid-but-wrong values |
| A customer sees the output directly | Free text or an answer | Reputation, refunds, or harm, with no internal check | Which cases are in scope; when to refuse or hand off |
The ticket example below uses a hybrid of the two: the model’s queue is applied automatically when the model is confident, and otherwise the ticket goes to a person. The choice follows from the workflow, which tells the team where a human can absorb errors, and that is the budget the rest of the criteria spend.
Draw the task boundary
A task boundary is the written statement of what the AI component is asked to do, on which inputs, producing which outputs, and what it must do when the input falls outside that statement. It is the contract between the application, the evaluation, and the baseline. Without it, “routing accuracy” can mean different things to the developer who writes the prompt, the labeler who builds the test set, and the manager who reads the dashboard.
A usable boundary fits on one page. For the ticket example:
Task: Assign one queue to each incoming support ticket.
Unit of input: One ticket (subject and body text, English, up to 2,000 characters).
Output: Exactly one of: billing, access, outage, howto, security.
Label rule: The queue that should handle the most urgent issue in the ticket.
Urgency order: security > outage > access > billing > howto.
Out of scope: Empty text, attachments only, languages other than English,
text over the length limit.
Out-of-scope Send to the human triage queue; do not guess.
behavior:
Abstention: The component may return no label; the ticket then goes to
human triage.
Ground truth: The queue assigned by a senior agent following this page,
recorded before the model's answer is shown.Each line removes an ambiguity that would otherwise surface later as a disagreement about whether the system works.
The unit of input decides what a single correct answer is. If a ticket is one message, a label is natural. If a ticket is a thread of twelve messages in which the problem changes, the unit might be the latest message or the whole thread, and the same system would receive different scores depending on the choice. Fix it now.
The label rule decides what correct means for the ambiguous cases. Roughly one ticket in twenty mentions two problems, for example a billing question and a service that is down. Without a precedence rule, either queue is defensible, and every disagreement between the model and the test labels becomes an argument rather than a measurement. A precedence rule makes the labeler’s job well defined. It also encodes a business decision: here the most urgent issue wins, because delay is most expensive there.
The closed output set is a design decision with consequences. A fixed list of five labels can be checked by a program, compared across systems, and counted. If the real work needs “none of these”, the list needs an explicit value or an abstention, because a closed list with no escape forces a guess on tickets that belong nowhere.
The ground-truth line decides what the evaluation can know. The labels come from people, and people disagree. If two agents following the page assign different queues to 8% of tickets, then agreement with a single agent’s labels cannot exceed what that agent’s own consistency allows, and differences between systems that are smaller than that noise cannot be read as differences in quality. Measuring agreement between labelers, as described in the entry on inter-annotator agreement, is part of defining the task, not a later cleanup step.
The kind of output also changes how hard it is to state the boundary. The table lists common task types and what “correct” means for each.
| Task type | Example | What correct means | How easily it is checked |
|---|---|---|---|
| Classification into a closed set | Choose a queue | The label equals the reference label | Automatically, if the reference exists |
| Extraction | Pull an invoice number from a message | The value equals the reference, or normalizes to it | Automatically, with rules for formats and absent values |
| Ranking or selection | Pick the three most relevant help articles | Relevant items appear near the top | Automatically with relevance judgments, which are costly to create |
| Free-text generation | Draft a reply | A reader judges it accurate, complete, and appropriate | Needs rubrics and human or model judges; no single reference string |
| Action selection | Decide to issue a refund | The action is allowed, correct, and has the right effect | Needs the downstream system or a simulator |
The further down this table a task sits, the more of its boundary has to be written as rules about acceptable behavior rather than as a single right answer. When a free-text or action task contains a sub-decision that has a closed answer, such as the queue, extract that decision and give it its own boundary, because that is the part that can be measured precisely.
Build the non-AI baseline first
A baseline is the reference point against which a candidate is judged: the best simple method that a team would deploy if the AI component did not exist. The glossary entry on baseline stresses that a baseline needs a stated population, period, and measurement procedure to be interpretable; the same holds here. It does three jobs.
It tells you what the AI component adds. A 94% accurate model is impressive only until a rule set reaches 91%. It is also the fallback when the AI component is slow, unavailable, or declines to answer, so it is a production component and not only a benchmark. And sometimes it is the answer: Zinkevich’s Rules of Machine Learning opens with the advice not to be afraid to launch without machine learning, on the reasoning that a heuristic can capture a large share of the available gain.
Baselines form a ladder, and a fair comparison goes at least up to the first rung that a motivated engineer could build in a day.
- The current process, measured. What it costs and how often it errs today, on the same task definition. If the process is a person, the baseline includes that person’s time.
- A trivial predictor. Always predict the most common class. Its score is the floor that any classifier must clear, and it shows how much of a headline number is prevalence.
- A rule set. Keywords, regular expressions, lookups against known senders or products.
- A small trained model. A classical classifier on simple features, when labeled data exists.
- The AI candidate.
The code below builds rungs 2 and 3 for the ticket task. The generator produces synthetic tickets from hand-written phrasings, some of which contain a typical keyword and some of which describe the same problem in other words, and about 6% of tickets add a second unrelated problem. The rule set is ordered, so the first matching rule wins, and unmatched tickets go to the largest queue. The listing is an excerpt of tickets.py; the dataset generator and evaluation printing are omitted.
RULES = [ # ordered: first match wins
("security", ["phishing", "token", "someone else"]),
("outage", ["down", "outage", "times out", "error page"]),
("billing", ["invoice", "charged", "refund", "payment", "bill"]),
("access", ["password", "log in", "login", "sign-in"]),
("howto", ["how do i", "how to", "where can i", "is there a way"]),
]
def rule_baseline(text):
t = text.lower()
for label, kws in RULES:
if any(k in t for k in kws):
return label
return "howto" # default queue: the largest oneRunning the full script on 2,000 synthetic tickets (Python 3.12, standard library only, seeded random generator) produces the following output.
class counts: {'access': 495, 'billing': 481, 'howto': 692, 'outage': 141, 'security': 191}
majority-queue baseline (howto): accuracy 0.346
keyword-rule baseline: accuracy 0.806
class n precision recall
billing 481 0.959 0.736
access 495 0.976 0.735
outage 141 0.810 0.667
howto 692 0.672 0.949
security 191 0.878 0.754
confusion (rows = true, columns = predicted)
billing access outage howto security
billing 354 2 7 115 3
access 5 364 9 113 4
outage 1 0 94 46 0
howto 9 7 6 657 13
security 0 0 0 47 144
true=access rule=howto | Urgent: I cannot get into the dashboard anymore
true=security rule=howto | Hi, Someone changed my account email and I did not do it
true=access rule=howto | Hi, I cannot get into the dashboard anymore
true=access rule=outage | Hi, I forgot my password and cannot log in. Also, the service is down for our whole teamRead the output from the top. The majority baseline always answers “howto” and reaches 0.346 accuracy, which equals the share of how-to tickets. That is the floor. The keyword rules reach 0.806, a large gain over the floor, and a number that might look acceptable in a status report.
The per-class table shows what that number hides. Precision for billing, access, and security is high: when a rule fires, it is usually right. Recall is low for every class except how-to: only 0.754 of security tickets and 0.667 of outage tickets reach their own queue. The confusion matrix shows where they go. Of 191 security tickets, 47 landed in how-to, and the same happens to 115 billing tickets and 113 access tickets. Tickets that do not contain a rule keyword fall through to the default queue, which explains why how-to precision is only 0.672 even though its recall is 0.949. The four printed errors show three failure sources: a paraphrase with no keyword (“cannot get into the dashboard”), the default queue absorbing it, and the precedence order picking “outage” for a ticket that mentioned two problems.
These numbers describe this synthetic phrasing list; real tickets would give different values. What transfers is the shape of the evidence. The baseline’s weakness is not uniform: it is concentrated in two classes that are rare and expensive. Paraphrase is also exactly the variation that a language model handles well, which is a concrete hypothesis about where the candidate could earn its place. A baseline that is built and read this way gives the team something to aim at. The entry on baseline heuristics shows with a different example how the choice of baseline and metric together can change the verdict.
Two cautions apply to baselines. The baseline should get the same effort a fair competitor would: a rule set tuned on the test data, or one deliberately left weak, makes the candidate look better than it is. And the baseline must be evaluated on the same inputs, labels, and metric as the candidate, which is why the task boundary came first.
Set quality and latency targets from the consumer
With a task and a baseline in hand, the next step is to say what quality and speed are required, in terms the consumer’s workflow supports.
Quality as a set of per-class floors
A single accuracy number averages over classes in proportion to how common they are. Our tickets are 35% how-to and 10% security, so a system that fails every security ticket loses at most ten points of accuracy while failing the class that matters most. Quality targets for a classification task are better expressed per class, using two definitions. For class \(c\), precision is \(\mathrm{TP}_c / (\mathrm{TP}_c + \mathrm{FP}_c)\), the share of tickets sent to queue \(c\) that belong there, and recall is \(\mathrm{TP}_c / (\mathrm{TP}_c + \mathrm{FN}_c)\), the share of tickets that belong in \(c\) that arrive there. Here \(\mathrm{TP}\), \(\mathrm{FP}\) and \(\mathrm{FN}\) count true positives, false positives and false negatives for that class. The article on classification metrics and decision thresholds develops these metrics and their averaging across classes in depth.
The consumer decides which of the two matters per class. For the security queue, recall is the target, because a missed security ticket is the expensive event; precision matters less until false alarms overload the security team. For the how-to queue, precision matters, because it is where mistakes go to hide: if how-to tickets contain a share of billing and access problems, those customers wait behind low-priority work.
Latency as a distribution with a budget
Latency is the time between the consumer’s request and a usable output. For a call to a hosted model, it is the sum of several parts: waiting in the client’s own queue, the network round trip, any queuing at the provider, the time to process the input, the time to generate the output, validation of what comes back, and any retries. Generation time grows with the length of the output, so the output design is a latency lever. A routing label of a few tokens is much cheaper to produce than an explanation of why the ticket belongs in the queue, which is another reason to fix the output contract in the task boundary.
Because latency varies from request to request, a single number is not a target. The mean is dominated by typical requests and hides the slow ones. The Google SRE book makes this point in its chapter on service level objectives: averaging latencies hides a long tail, and high percentiles expose it. The 95th percentile (p95) is the value that 95% of requests are at or below. The glossary entry on percentile covers the calculation conventions, and the one on service level objectives covers how targets like these are stated and monitored. A latency target has three parts: the statistic (p95), the threshold (1.5 seconds), and the condition under which it is measured (end to end from the application, at the expected load).
For ticket routing, the budget comes from the workflow. The suggested queue should appear within 1.5 seconds of an agent opening the ticket; if it does not, the interface shows the keyword-rule result instead. The baseline thus doubles as the deadline fallback.
The simulation below shows why the percentile matters and why a chain of calls needs its own measurement. The per-call latency model is invented for this article: a network round trip of 30 to 60 ms, a log-normal model time with a 450 ms median, and a 3% slow path that adds an exponentially distributed delay with a 1.2 s mean. The parameters are not measurements of any service. Real values must be measured from the application under realistic load. The listing is an excerpt of latency.py.
import math
import random
import statistics
def one_call_ms(rng):
network = rng.uniform(30, 60) # round trip, ms
model = rng.lognormvariate(math.log(450), 0.5) # median 450 ms
if rng.random() < 0.03: # slow path: queueing, cold capacity
model += rng.expovariate(1 / 1200)
return network + model
def pct(values, p):
return statistics.quantiles(values, n=100, method="inclusive")[p - 1]
def summarize(name, xs, budget):
over = sum(x > budget for x in xs) / len(xs)
print(f"{name:22s} mean {statistics.fmean(xs):6.0f} p50 {pct(xs,50):6.0f} "
f"p95 {pct(xs,95):6.0f} p99 {pct(xs,99):6.0f} over {budget} ms: {100*over:4.1f}%")Running it with 50,000 simulated requests for one call and for two sequential calls gives:
one call mean 590 p50 503 p95 1176 p99 2084 over 1500 ms: 2.3%
two sequential calls mean 1181 p50 1067 p95 2065 p99 3363 over 1500 ms: 17.5%
p95(one call) x 2 = 2351 ms
p95(two sequential calls) = 2065 msThe single call has a mean of 590 ms and a median of 503 ms, so the typical request looks comfortably fast. Its p95 is 1,176 ms and its p99 is 2,084 ms, and 2.3% of requests exceed the 1.5 s budget. In this design, 2.3% of tickets show the fallback result. Whether that is acceptable is a product decision, and the number says exactly what is being accepted.
The second row models a workflow in which one call classifies the ticket and a second call, say, extracts fields from it. The means add: 590 ms becomes 1,181 ms. Percentiles do not add. The p95 of the two-call chain is 2,065 ms, less than twice the single-call p95 of 1,176 ms (2,351 ms), because a slow first call and a slow second call rarely coincide. But the chain exceeds the 1.5 s budget for 17.5% of requests, against 2.3% for one call. Adding each step’s p95 overstates the chain’s p95, and adding medians understates it, so the chain is measured end to end. Handling timeouts, retries and rate limits at the integration layer is the topic of the next article, on integrating model APIs; here the point is that the budget and its fallback are requirements to fix in advance. How a team later buys latency and cost with caching, routing and smaller models is covered in the article on optimizing quality, latency and cost.
Price the errors and write the acceptance criteria
Per-class floors say which errors matter. They do not say how much, or how to trade one against another. That requires a cost for each kind of error, which is what a cost matrix holds.
A cost matrix
Let \(y\) be the true queue, \(\hat y\) the queue the system chose, and \(c(y, \hat y)\) the cost of that combination, with \(c(y, y) = 0\). The expected cost per ticket is
\[ \bar{C} = \sum_{y}\sum_{\hat y} P(y, \hat y)\, c(y, \hat y), \]
where \(P(y, \hat y)\) is the share of tickets with true queue \(y\) that the system sends to \(\hat y\), weighted by how common \(y\) is. In words: weight each cell of the confusion matrix by its price and add. Estimating \(P\) from a labeled sample turns this into the mean of the observed per-ticket costs.
Costs can be money, staff time, or any common unit. The hypothetical costs below are in minutes of staff time. They are assumptions made for this example, not measurements. In a real project they come from the people who own the workflow, and the exercise of agreeing on them is itself valuable.
| Situation | Cost (minutes) | Reasoning |
|---|---|---|
| Correct queue | 0 | Nothing to repair |
| Ordinary ticket in the wrong queue | 6 | An agent notices and re-routes it |
| Any ticket wrongly sent to security or outage | 10 | A false alarm pages or interrupts someone |
| Outage ticket in a non-outage queue | 60 | The outage response is delayed |
| Security ticket in a non-security queue | 120 | A security issue sits unseen in an unrelated queue |
| A person triages a ticket | 3 | Reading time; assumed to be correct |
| A model call | 0.1 | Per-ticket cost converted to the same unit |
The cost function and the stand-in model go together in costs.py. The stand-in returns a label and a confidence for each ticket given its true label, with the per-class accuracy and confidence distributions set in the code, so its behavior is simulated; it takes the place of results from a real model run on a test set. The listing shows the first 26 lines of the file.
import random
from tickets import LABELS, make_dataset, rule_baseline
# ---- stand-in for model outputs (not a real model) -------------------------
RECALL = {"billing": .96, "access": .95, "outage": .90, "howto": .96, "security": .91}
def stub_model(label, rng):
"""Return (predicted_label, confidence) with per-class accuracy set above."""
if rng.random() < RECALL[label]:
return label, rng.betavariate(8, 1.5)
others = [l for l in LABELS if l != label]
return rng.choice(others), rng.betavariate(3, 3)
# ---- hypothetical cost matrix, in minutes of staff time ---------------------
def cost(true, pred):
if true == pred:
return 0.0
if true == "security":
return 120.0 # a security ticket waiting in the wrong queue
if true == "outage":
return 60.0 # an outage ticket waiting in the wrong queue
if pred in ("security", "outage"):
return 10.0 # false alarm that pages someone
return 6.0 # ordinary re-route
HUMAN_TRIAGE = 3.0 # a person reads the ticket and routes it correctly
MODEL_CALL = 0.1 # per-ticket model cost expressed in the same unitApplying the cost matrix first to the baselines, and then to the stand-in at several abstention thresholds on 400 synthetic tickets, produces:
manual triage of every ticket: 3.00 per ticket
keyword rules: 3.37 per ticket
threshold abstain % accuracy cost/ticket
0.0 0.0 0.943 2.48
0.3 1.0 0.952 2.14
0.4 2.0 0.962 1.81
0.5 4.0 0.971 1.25
0.6 6.2 0.976 1.00
0.7 12.5 0.986 0.83
0.8 33.2 0.993 1.26
0.9 67.2 1.000 2.12
lowest cost at threshold 0.7: 0.83 per ticket
cost share by true class (no abstention):
billing 54.0 5.7%
access 36.0 3.8%
outage 240.0 25.2%
howto 22.0 2.3%
security 600.0 63.0%Three observations follow from this output.
First, accuracy and cost can disagree. The rule set has 0.806 accuracy on the earlier dataset, yet its cost on this sample is 3.37 minutes per ticket, more than the 3.00 minutes that manual triage costs. On this sample, keeping the manual process would be cheaper than deploying the rules, because the rules’ mistakes land on the two expensive classes.
Second, the cost of the stand-in without abstention (threshold 0.0, accuracy 0.943) is 2.48 per ticket, and 63% of that is security tickets and another 25% outage tickets, which together are under a fifth of traffic. The last block of the output makes the point that most of the cost of a 94% accurate system sits in a small number of rare, expensive errors. Improving overall accuracy by fixing frequent cheap mistakes would barely move the cost.
Third, abstention changes the picture. When the confidence is below a threshold, the ticket goes to a person at the price of triage plus the model call. The cost per ticket falls from 2.48 to 0.83 at a threshold of 0.7, where 12.5% of tickets are handed to people and accuracy on the rest rises to 0.986. It rises again for higher thresholds, to 2.12 at 0.9, because so many tickets go to people that their cost dominates. The curve is U-shaped for a reason that can be stated directly. Raising the threshold removes errors that cost more than a human review and then starts removing correct answers that did not need one.
Where the thresholds come from
For a binary decision, the choice reduces to a one-line rule. Suppose the system holds a probability \(p\) that a ticket is a security ticket. Sending it to security costs \((1-p)\,c_{FP}\) in expectation, where \(c_{FP}\) is the cost of a false alarm. Not sending it costs \(p\,c_{FN}\), where \(c_{FN}\) is the cost of a miss. Send it to security when
\[ p \;\ge\; \frac{c_{FP}}{c_{FP} + c_{FN}}. \]
With the costs above, \(10 / (10 + 120) \approx 0.077\). The system should flag a ticket as security even when it is more than 92% likely to be something else, because a miss is twelve times worse than a false alarm. This only holds if \(p\) is a calibrated probability, that is, if tickets scored 0.2 are security tickets about 20% of the time; the article on probability calibration covers how to check that. A model’s stated confidence, or a score derived from token probabilities, is not calibrated by default. In the stand-in, confidence is used only as an abstention score, and the useful property is that wrong answers tend to have lower confidence than right ones, which should be verified for any real model rather than assumed.
Abstention extends the rule. If a person handles a ticket at cost \(h\), the system should answer by itself only when the cheapest available answer has expected cost below \(h\), and defer otherwise. A cost-based threshold is therefore a consequence of the cost matrix, not a tuning knob picked by taste.
Acceptance criteria
Acceptance criteria are conditions, fixed before the evaluation, that decide whether the candidate ships. Each criterion has five parts: the metric and how it is computed, the data it is computed on, the target, the rule for handling statistical uncertainty, and the action taken on each outcome. Criteria written as “good accuracy” or “fast enough” fail on every part.
The uncertainty rule is needed because a metric measured on 78 security tickets is an estimate, not the true rate. For a proportion, such as recall, the Wilson score interval gives a range of plausible true values. With \(k\) successes in \(n\) trials, \(\hat p = k/n\), and \(z = 1.96\) for a 95% interval, its endpoints are
\[ \frac{\hat p + \dfrac{z^2}{2n} \;\pm\; z\sqrt{\dfrac{\hat p(1-\hat p)}{n} + \dfrac{z^2}{4n^2}}}{1 + \dfrac{z^2}{n}}. \]
The interval stays within 0 and 1 and behaves sensibly when \(\hat p\) is near 1, which the simple symmetric formula does not; the entry on the Wilson score interval explains the construction, and the NIST handbook gives formulas and worked examples. For quantities that are means of costs, which have no simple formula, the bootstrap resamples the observed per-ticket costs to estimate the same kind of range.
With intervals, a criterion has three outcomes, not two. It passes when the whole interval is on the right side of the target, fails when the whole interval is on the wrong side, and is inconclusive when the interval straddles the target. An inconclusive result does not mean “probably fine”; it means the sample cannot decide.
The program below tunes the abstention threshold on a separate development set of 600 synthetic tickets, evaluates once on a held-out set of 800, and prints one row per criterion. Tuning on the same data used for the final claim would select a threshold that flatters the result, so the two sets are separate. The listing is an excerpt from gate.py, which also imports the stand-in, the cost function and the latency model.
def wilson(k, n, z=1.96):
p = k / n
d = 1 + z * z / n
centre = (p + z * z / (2 * n)) / d
half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
return centre - half, centre + half
def run(items, tau, rng):
"""End-to-end outcome per ticket: (true, final_queue, answered_by_model, cost)."""
out = []
for text, label in items:
pred, conf = stub_model(label, rng)
if conf < tau:
out.append((label, label, False, HUMAN_TRIAGE + MODEL_CALL))
else:
out.append((label, pred, True, cost(label, pred) + MODEL_CALL))
return out
def mean_cost(out):
return sum(o[3] for o in out) / len(out)
def bootstrap_ci(values, rng, reps=2000):
n = len(values)
means = sorted(sum(rng.choices(values, k=n)) / n for _ in range(reps))
return means[int(0.025 * reps)], means[int(0.975 * reps)]
def verdict(lo, hi, target, higher_is_better):
if higher_is_better:
return "PASS" if lo >= target else ("FAIL" if hi < target else "INCONCLUSIVE")
return "PASS" if hi <= target else ("FAIL" if lo > target else "INCONCLUSIVE")Its output:
dev cost per ticket by threshold: {0.0: 2.21, 0.3: 2.08, 0.4: 1.78, 0.5: 1.61, 0.6: 1.06, 0.7: 0.81, 0.8: 1.34, 0.9: 2.33}
chosen threshold: 0.7
cost per ticket | 0.71 [0.50, 1.05] | <= 2.00 | PASS
security reaches right queue | 77/78 = 0.987 [0.931, 0.998] | >= 0.97 | INCONCLUSIVE
outage reaches right queue | 59/59 = 1.000 [0.939, 1.000] | >= 0.95 | INCONCLUSIVE
answered by model | 687/800 = 0.859 [0.833, 0.881] | >= 0.70 | PASS
latency p95 (ms) | 1176 | <= 1500 | PASS
same observed security recall (99%) at different sample sizes:
n= 100 99/100 interval [0.946, 0.998] vs target 0.97 -> INCONCLUSIVE
n= 300 297/300 interval [0.971, 0.997] vs target 0.97 -> PASS
n= 1000 990/1000 interval [0.982, 0.995] vs target 0.97 -> PASS
n= 3000 2970/3000 interval [0.986, 0.993] vs target 0.97 -> PASSThe development set picks the same threshold, 0.7. On the held-out set, three criteria pass. The cost per ticket is 0.71 with an interval of 0.50 to 1.05, whose upper end is below the 2.00 target. The model answers 85.9% of tickets without a person, with a lower bound of 0.833 against a 0.70 target. The p95 latency is 1,176 ms against 1,500 ms.
Two criteria are inconclusive. Of 78 security tickets, 77 reach the right queue, a rate of 0.987, but the interval is 0.931 to 0.998 and its lower end is below the 0.97 target. The outage criterion is the more instructive row: 59 of 59 correct is 100% observed, and the interval still extends down to 0.939, below the 0.95 target. A perfect score on 59 tickets does not establish a 95% floor.
The last block shows what would change the verdict. If the observed security rate were 99%, then 100 tickets would still be inconclusive (interval 0.946 to 0.998), while 300 would pass (lower bound 0.971) and 1,000 would pass comfortably. The numbers depend on the assumed rate; with a true rate closer to the target, the sample needed grows rapidly. Because security tickets are about 10% of traffic, getting 300 of them from a random sample takes about 3,000 tickets, and a team that cannot afford that has three honest options: oversample the rare class when building the test set and report class-level results separately; reduce the claim, for example to a recall target of 0.95; or change the design so that the model never decides security alone, for example by sending every ticket the model rates as possibly security to a person. These are decisions about the task and the workflow, which is why acceptance belongs to the framing and not to the end of the project.
Two more things belong in the criteria beyond the table. The first is the delivered behavior when parts of the system fall back. The 2.3% of tickets that miss the latency budget get the rule-based answer, so the cost people experience is a blend. Assuming, for simplicity, that fallback is independent of the ticket content, it is about \(0.977 \times 0.71 + 0.023 \times 3.37 \approx 0.77\) minutes per ticket, which is still well inside the target but is the number to report. The second is the freeze. The criteria, the test set, the prompt and the model version are fixed before the run, and a change to any of them restarts the evaluation. Criteria adjusted after seeing results become descriptions of what happened, and they no longer decide anything.
Offline acceptance shows that a candidate is worth trying on live traffic. It does not show that users or workflows benefit, because exposure, workload, and how people respond to suggestions change what a metric means in production; the article on offline metrics and online outcomes examines these gaps. Building a test set that is representative, labeled consistently, and reusable is the subject of the article on reproducible test sets, and staged exposure and rollback criteria are covered in the article on online evaluation and controlled rollout.
Recording the decision
The framing is complete when it fits in a short record that every later stage can refer to. For the ticket example:
Consumer: Queue system; agents work tickets from the assigned queue.
Task boundary: One ticket in, one of five queues out, or abstain; see task card v1.
Baselines: Manual triage (3.0 min/ticket); keyword rules v1 (3.37 min/ticket
on the evaluation set).
Costs: Cost matrix v1, in minutes of staff time.
Acceptance (all must PASS on the frozen test set, 95% intervals):
- Cost per ticket, upper bound <= 2.00
- Security and outage recall, end to end, lower bound >= 0.97 and >= 0.95
- Share answered without a person, lower bound >= 0.70
- p95 latency, application to application, <= 1.5 s; fallback share reported
Inconclusive: collect more labeled tickets of the affected class; do not ship.
Fail: Keep the baseline; revise the design or the candidate.Everything in this record can be turned into a test that runs on every candidate. If the baseline passes, the right result is to ship the baseline. A candidate earns its place by passing criteria that a cheaper alternative fails, and the criteria are written so that anyone can see which one it did not meet.
References
- Sculley, D., et al. “Hidden Technical Debt in Machine Learning Systems.” Advances in Neural Information Processing Systems 28 (NIPS 2015). https://papers.nips.cc/paper_files/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
- Zinkevich, M. “Rules of Machine Learning: Best Practices for ML Engineering.” Google for Developers. https://developers.google.com/machine-learning/guides/rules-of-ml
- Jones, C., Wilkes, J., Murphy, N., with Smith, C. “Service Level Objectives.” In Site Reliability Engineering (Google). https://sre.google/sre-book/service-level-objectives/
- NIST/SEMATECH e-Handbook of Statistical Methods, “7.2.4.1. Confidence intervals” (confidence intervals for proportions, including the Wilson method). https://www.itl.nist.gov/div898/handbook/prc/section2/prc241.htm
- scikit-learn User Guide, “Metrics and scoring: quantifying the quality of predictions” (confusion matrix, precision, recall, dummy estimators, custom scorers). https://scikit-learn.org/stable/modules/model_evaluation.html
- Python documentation, “statistics — Mathematical statistics functions” (quantiles and their interpolation methods). https://docs.python.org/3/library/statistics.html
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.