Data Quarantine

Data quarantine separates rejected records from accepted output while preserving enough evidence to investigate and repair them. Keep the source or batch identity, rejected input under its access policy, validation reason and relevant rule version. A count of failures without recoverable records is insufficient for repair.

If one of 100 orders has an invalid amount, the rejection rate is 1%. That does not make the other 99 a complete financial total: the missing order may carry most of the value. Define whether incomplete publication is allowed and whether unknown impact itself requires holding the batch. Key or schema failures may invalidate the whole input rather than one row.

If bad data has reached production, first contain the affected path according to its impact: hold publication, isolate the affected output, or pause the consumers that would spread it. Determine which periods, keys, and downstream results were affected using published versions, source snapshots, processing history, lineage, and reconciliation. Quarantine records provide evidence about rejected input, but records that incorrectly passed validation may never have entered quarantine. Rejected rows alone therefore cannot bound the damage. Repair and reprocess the identified scope, verify the resulting outputs, and notify consumers that received incorrect results.

Give corrected input a new version or batch identity linked to the rejection. Do not silently change the input behind an ID already treated as immutable. Apply accepted rows with retry-safe writes and preserve whether the published result is partial. Quarantine is a managed recovery path, not an indefinite discard folder or proof that the remaining rows are correct.

Further reading

• AWS idempotent request identity