Shadow Testing

Shadow testing sends copies of selected live inputs to a candidate while the existing system continues to provide the result used by the user. It helps examine candidate behavior on operational inputs without directly substituting its answer for the production answer.

Copying an input must not duplicate an action

Imagine a returns assistant receiving a request to submit a return. The live assistant may call the order service. If its shadow also submits the return, the test has created a second real action even though the shadow answer is hidden.

Disable, mock, or isolate side effects in the shadow path. Confirm that copied data can be used in that environment and that additional computation does not exhaust shared capacity. A shadow system still consumes resources and handles potentially sensitive information.

Real inputs do not provide every kind of evidence

Compare outputs against task criteria and inspect failures, latency, and dependency behavior. The current answer is a comparison reference, not necessarily ground truth.

Because users do not act on the shadow answer, the test cannot directly measure how that answer would change user behavior. Differences in resources or isolated dependencies can also limit performance comparisons. Record those differences when interpreting the result.

Reference: Amazon SageMaker AI: Shadow tests.