Data reconciliation (DR) is a term typically used to describe a verification phase during a data migration where the target data is compared against original source data to ensure that the migration architecture has transferred the data correctly.
During a data migration, it is possible for mistakes to be made in the mapping and transformation logic. Also, runtime failures such as network dropouts or broken transactions can lead to data being left in an invalid state. These problems can lead to a range of issues such as:
Without the data reconciliation stage, these issues can go unnoticed, severely damage the overall accuracy of your data and lead to inaccurate insights and issues with customer service.
The traditional approach to data reconciliation has often relied on simple record counts to observe whether the expected number of records had been migrated. This was typically due to the processing power required to perform field-by-field validation. However, the issue with this is that missing records is just one mistake that can arise from a data migration (as you can see above). The other issues will therefore go unnoticed.
Modern data migration solutions (such as Aperture Data Studio) therefore provide data prototyping functionality and comparative reconciliation capabilities that enable full volume data reconciliation testing – that help identify where mistakes such as duplicate records have occurred.