Reconciliation

In data engineering, reconciliation means comparing data with an agreed reference and investigating differences. The reference might be a source extract, a file manifest, or an independently produced total. Its scope and cutoff must match the data being checked.

Equal counts can hide different records

Suppose the reference contains orders A, B, and C for \$10, \$20, and \$30. The target contains A, B, and D for the same amounts. Both have three rows and a \$60 total, yet C is missing and D is unexpected.

A count catches some omissions. A total catches some value differences. Comparing identifiers reveals the mismatch in this example; comparing values by identifier can then find altered amounts. AWS DMS documents row-level source-to-target validation as one concrete implementation of comparison.

Decide what equality should mean

If the target aggregates orders by store, direct row equality with individual orders is inappropriate. First apply the agreed grouping and filters to the reference. Align time zones, currencies, refund treatment, and the cutoff for late arrivals. Record why any numerical tolerance is acceptable instead of choosing one merely to make a check pass.

A discrepancy is evidence to investigate, not an instruction to overwrite the target immediately. The reference may itself be incomplete or may describe a different snapshot. Reconciliation supports confidence in a defined relationship; it cannot establish truth when both sides share the same error.

Reference: AWS DMS: Data validation.