The continued need for impact assessment in data migration

I was delighted to be invited to take part in the recent Experian Data Migration Roundtable in London last month. At the event, we saw two very clear themes emerge. Firstly, the need for an impact assessment before commencing a data migration project and secondly, on using data migration as a design for data protection. So in this first part I’d like to kick off with a detailed look into our discussions around impact assessment and next week I'll tackle the issue of data migration as a driver for data protection by design.

Experian Data Quality invited a great cross-mix of practitioners from varying sectors. It was interesting to hear common tales from both the commercial and public sector contributors, working in all sorts of industries, from finance to not-for-profit, telecoms to local government.

A recurring theme was the need to de-risk data migration projects and also reduce the failure rate. “We’ve experienced significant delays and failure in the past” was a recurring remark.

This is a common issue and it was one of the reasons I focused my presentation on the need to create a Pre-Migration Impact Assessment prior to the migration kick-off. Data migration projects are risky affairs at the best of times, but you can create undue risk (and stress!) by failing to assess the viability of your migration from the outset.

The key to risk management is the removal of assumptions and as any seasoned data migration practitioners will testify, migration projects are often rife with assumptions.

We had an interesting section in the roundtable where several of us recounted some of the migration assumptions we’ve witnessed in the past. Derek Munro came up with some great practical examples such as:

• Hidden Nulls: A field always has a value, but the value is often just a space filler, such as the word “null”
• Unexpected relationship: An example given was a contract must only have one address, yet the system has some which has two addresses
• Duplicate identifiers: You’re told it’s impossible to have duplicate identifiers yet the users find creative ways to get them in!
• Overloaded fields: Particularly in older systems, you may find examples of overloading, a condition where the users find that the system can’t cope with a new data item they need to store or task they need to perform, so they add important information to a comments field or append special indicators to code fields
• Manual linkage: Two systems need to be linked/joined, yet there is no “common key” – the business process relies on people to study the values/attributes in each system and decide on which records are linked based on “gut feeling”.

These were some of the technical assumptions discussed but more strategic assumptions were outlined as well. I talked of one client who had no idea of how many systems were involved in their migration but they assumed it could be completed in twelve months. Indeed, assuming the project was even feasible, given their technical strategy, was also incorrect. The subsequent pre-migration impact assessment discovered major flaws in their data quality, leading to a total re-think in terms of the technical strategy for migration.

Suffice to say, if you are planning a future migration then the pre-migration assessment is possibly the single most beneficial activity I can recommend for accurately scoping and de-risking your project because it clears away all assumptions and substitutes known facts. These facts help you plan and cost your migration effectively. They tell you what tools are required, what skills and strategy to adopt, what the likely pitfalls will be.

The Pre-Migration Impact Assessment is effectively a simulation of the whole migration, completed in a rapid timeframe. I explained how modern technology facilitates this migration simulation concept and Jim Williams duly took up the mantle by demonstrating how tools like Experian Pandora completed tasks like disparate relationship discovery, automated rule assessment and migration prototyping.

Technology aside, the lesson here is a simple one. You wouldn’t leap over a wall without knowing what’s on the other side. A Pre-Migration Impact Assessment gives you the confidence of what to expect when you make the leap on your migration project.

There were lots of nodding heads following an explanation of the impact assessment technique, so hopefully people will get the chance to put the concept into practice and reduce those risks and failures that are so common in the migration industry.

What are your thoughts on the concept of a Pre-Migration Impact Assessment? Please share your views below and don't forget to read my post next week where we explore the issue of data migration as a driver for data protection by design.