Behavioral Testing

Behavioral testing evaluates a model by checking how it behaves on designed inputs, organized by the capability being tested, instead of summarizing its quality in one accuracy number on a held-out set. The model is treated as a black box: the tests state what any acceptable system must do, and do not depend on how the model works internally.

Why held-out accuracy is not enough

An aggregate score hides where the model fails. A test set drawn from the same distribution as the training data shares the training data's shortcuts, so a model that relies on a spurious cue scores well and breaks when the cue disappears. A single number also cannot tell a user whether the model handles a negation, a paraphrase, or an irrelevant added sentence. For free-text outputs there is a second problem: comparing against a stored exact string breaks on harmless variation and can pass on harmful variation that happens to match.

Ribeiro, Wu, Guestrin and Singh proposed CheckList (ACL 2020), borrowing the idea of behavioral testing from software engineering. They organize tests by capability, such as vocabulary, negation or robustness, and by the kind of expectation, and in user studies found more bugs with it than without it.

Three kinds of test

• Minimum functionality test (MFT). Simple inputs with an unambiguous correct answer, like unit tests: 10 days since purchase with a 30-day return window must give "eligible"; 45 days must give "not eligible". They show that the basic capability exists.
• Invariance test (INV). Change the input in a way that must not change the output: paraphrase the question, add an irrelevant sentence, reorder a list of evidence, change a person's name. A different output exposes dependence on surface features.
• Directional expectation test (DIR). Change the input in a way that must move the output in a known direction: shrink the return window and a borderline case must flip from eligible to not eligible; make a review more negative and the sentiment score must not rise. These expose a model that ignores the evidence and answers from habit.

A test case is an input, a perturbation (if any), and an expected relation (equals this label, equals the unperturbed output, moves in this direction). Many cases are generated from templates, so that one template with a list of fillers gives hundreds of cases and a failure rate per capability.

Example (hypothetical toy model)

A toy function decides return eligibility from the number of days and the text of the question, and has a flaw: it treats the word "broken" as grounds for eligibility regardless of the window. Running the suite:

type	input	expected	got
MFT	10 days	eligible	eligible
MFT	45 days	not eligible	not eligible
INV	10 and 45 days, paraphrased question	same as unperturbed	same
INV	10 and 45 days, with "It arrived on a Tuesday." added	same as unperturbed	same
DIR	10 days, then 35 days	eligible, then not eligible	as expected
INV	45 days, with "My item is broken." prepended	not eligible (the window has passed)	eligible, FAIL

Eight of nine cases pass, so an accuracy-style summary would look healthy, and the single failure is exactly the capability that matters: the model answers from a cue and not from the window.

Running them on a language model

Language-model output is variable, so a behavioral test checks a property of the output rather than a string: the decision label is one of the allowed values, the cited source ids are a subset of those supplied, the answer abstains when the evidence is missing, injected instructions in a document are ignored. Because one run is a sample, run each case several times and report a pass rate. Set the pass criterion before the run, for example "no failures on MFT cases, at most 2% on INV cases", and compare the candidate and the current release on the same cases with the same number of runs. These tests cost model calls, so they sit in a layer above fast deterministic tests of the code that builds the prompt.

Limits

• A suite covers the capabilities its authors thought of. It finds the bugs it is designed to find and not others, and it is not a proof of correctness.
• Invariance depends on a judgment about what should not matter. A paraphrase that changes meaning, such as swapping "at least" and "at most", is not a valid invariance test, so perturbations need review.
• Pass rates on small template sets are estimates; report the number of cases next to the rate.
• Failures should feed back into the suite. A bug found in production becomes a new case, so it cannot return unnoticed.

Neighbors

Held-out evaluation measures average quality on realistic data and complements behavioral tests; the two answer different questions. Metamorphic testing in software testing uses the same idea of checking relations between outputs of related inputs. Adversarial testing searches for worst-case inputs, while behavioral tests check specified behaviors.

References

• Ribeiro, Wu, Guestrin and Singh, Beyond Accuracy: Behavioral Testing of NLP Models with CheckList