Technology is constantly evolving and business needs are always changing. Given this state of consistent change, a majority of organizations will need to take on a data migration project at some point. In our recent 2017 global data benchmark report, we found that 35 percent of organizations will start a data migration this year. Data migrations are costly projects: they require a good deal of planning, and involve a number of stakeholders from across your business. A data migration is typically one step in launching a new system of some sort—be it a customer relationship management platform, an advanced business intelligence (BI) reporting tool, or consolidation of multiple sources into a data warehouse. After taking the time to vet the various options and investing a significant amount of money into a shiny new system, focusing your time and resources to preparing the data for a migration can often feel a little less exciting. While it is not cheap, the cost of a data migration is only a fraction of the money you may spend on purchasing and implementing a new system—but a successful data migration is critical to set your business up for success.
Let’s think of your new system as a new house. After you buy a new house, you are likely going to pack up most of your belongings and take them with you. Moving is often a chance for people to go through their stuff and decide what is worth bringing to the new house, and what things might be ready for the trash. If you still have that A+ dinosaur diorama from the second grade, a move might be a good time to evaluate whether it’s worth bringing that with you or not. It may also be a chance to decide that the old couch in your garage may not fit so well in your newly finished basement. Your data migration is similar to the moving process. If your antiquated, out-of-date stuff wouldn’t make it into your new house, why would you be willing to migrate old or poor quality data into your new system?
83% of data migrations fail or exceed their budgets and schedules
Data migrations can be tricky. These projects hold many challenges, and according to Gartner, 83 percent of data migrations fail or exceed their budgets and schedules.1 Though it’s unfortunate that so many organizations experience failure, the good news is that we can learn a great deal from their mistakes. We have identified some of the most common pitfalls that companies run into. Whether you are currently migrating your data or plan to at some point in the future, beware of the biggest obstacles that may stand in your way of success:
Data quality – According to our study, 44 percent of U.S. organizations say that data quality issues caused delays for their migration projects. Data quality is essential to a successful data migration. When organizations undertake a data migration, they often unearth data quality issues that may not have been noticeable in their existing systems because of workarounds or system shortcomings, but that become glaring when it’s time to switch to a new system. Poor data quality slows down the entire migration process, and sometimes bad data will be outright rejected by the new system as newer technologies often have stricter standards surrounding the quality of data entering the system.
Lack of standardization – To easily switch from an old system to a new one, data should be formatted the same way. Standardization is important to ensure that all of the data you are migrating from one or more existing systems is in the correct format to be understood in the new system. If, for example, the new system requires that names be listed as (last, first) and you have a mix of formatting for your existing data, that lack of standardization can create significant issues and slow down the migration process.
Insufficient labor – Many organizations rely heavily on manual processes when conducting a data migration. This entails teams of SQL experts coding queries to extract all the existing data and prepare it for migration—often without understanding the business context or the state of data quality. Considering that migrations involve moving hundreds of thousands, if not millions of records, the amount of manpower required is immense. We found that for 29 percent of organizations, insufficient labor was a major obstacle that delayed their data migrations. Even for organizations that have the software and tools needed to process the data, successful data migrations still involve a great deal of effort on the behalf of the IT team and many other stakeholders across the business.
Not enough time - As mentioned earlier, another big issue that many organizations encounter is that they run out of time. A data migration is often a timely process, and when companies do not proactively take steps to analyze their data at the outset, they are often surprised at the sheer volume of information that must be migrated. Many organizations also fail to build in time for an initial analysis or the many user acceptance tests that are typically conducted to see whether the data works in its new environment. Discovering that you have more data than expected or finding that the data fails in the new system can significantly delay the data migration time frame. When a user acceptance test fails, for example, the team must start over again, build something new, and test it, repeating the process until something works. Often, the migration timeline is determined by deadlines imposed by the specific business objective driving the adoption of a new system.
Ineffective tools – There are many different solutions and tools that organizations use to complete a data migration. Some organizations require robust, comprehensive tools, but many organizations can combine some lighter-weight tools that are less expensive. Often, the tools designed specifically for the migration lack data quality capabilities that are essential to success. Ensuring that you have a data migration tool to address data quality needs can help make your migration more effective and more efficient.
Poor communication – A lack of communication among the stakeholders in a data migration is common and can lead to tremendous issues. In our study, more than half of the respondents (51%) agreed that improved communication would have helped to reduce delays in the process. Generally, the IT and business teams will meet at the beginning of the process to discuss scope and expectations of the new system, but then IT will handle the full migration with minimal involvement from business users across the organization. Without business input throughout the process, IT completes the required technical specifications of the migration, but has little sense of whether or not the results will be practical and useful for the business at large.
Budget restrictions – Two of the most common reasons why organizations’ data migrations fail are that they run out of money or run out of time. This is because companies frequently overlook the budgeting and forecasting stage of the migration. Investing the money it takes to successfully complete a migration is necessary for achieving the desired outcomes or any potential return on investment from the new system. Many organizations end up setting themselves up for failure by never determining what an accurate budget might look like and underestimating how much a data migration might cost. Many businesses also ignore best practices, and their costs end up being higher in the end. Any perceived savings an organization tries to achieve by skimping on planning and preparatory investments ends up coming back to bite them later on, and can often have repercussions beyond just the budget.
Despite so many potential obstacles, a well-prepared organization can achieve a successful data migration. Planning effectively and getting the right people, processes, and tools in place can help you overcome these challenges and deliver the migration on time and on budget. Following the best practices outlined in the next section can help you avoid these pitfalls and set you up for a more seamless data migration.
If there is one true secret to data migration success, it’s all in the planning. Planning for your data migration can be an intensive process, but it will create a better experience for all involved, and it’s the best way to ensure your new system will perform as desired. Knowing your specific needs and what you wish to get out of your data migration will help you understand the right team of people to put together, what your process will look like, and which tool or combination of tools will adequately fit your needs.
There are two main approaches to data migrations: big bang or phased.2 In some instances, organizations require the big bang approach and need to transfer all their data to the new system, or series of systems in one fell swoop. This approach provides the benefit of a shorter timeline, which can potentially lead to lower costs. The big bang approach is often necessary for organizations that don’t have the option of simultaneously working in both systems and can’t afford any downtime for normal business operations. Organizations often plan for big bang migrations over the course of a number of months, and then execute the full migration over a weekend or holiday. The phased approach moves data in smaller segments over a longer period of time. The process involves less risk and provides ample time for business users to learn and acclimate to the new system, which can help with adoption. Sometimes organizations divide their migration and utilize both approaches, so perhaps a large chunk of essential business data is migrated via big bang, while other elements are migrated in a phased approach.
The planning period is your chance to determine the roadmap for your migration and to choose the course that will be most effective in terms of your desired outcome, timeline, and budget. As with most planning, it is important to have the end in mind. A data migration is not as simple as just moving information from point A to point B. You need to define what success will look like in the target system. What is necessary to ensure the new system operates optimally? What degree of data quality do you expect in the new system? How can you be certain that you achieve the required level of data quality? How much data needs to migrate to the new system? What data can you filter out from the old system? Asking these questions can help you determine a more detailed scope for the migration. The more questions and variables you consider at the outset, the better.
After you’ve defined what the end should look like, it’s important to know where you are now. Defining your current state of data quality can help you get a sense of what a realistic goal is for the target system. If your current state of data quality is poor and you hope to have a high degree of data quality in the new system, you can build in the time and budget to improve your data quality as part of the migration. You also will want to analyze your data sources and compare what the source data looks like—in terms of organization, formatting, grouping, etc.—to how the data will look in the target system. If, for example, you currently have customer information on one platform and business performance information on another, and you plan to host both types of data in a new system, the way the data is organized and grouped may look different.
Another very important thing to keep in mind is that with a data migration, less really is more.
Another very important thing to keep in mind is that with a data migration, less really is more. The less data you transfer from the existing system, the less it will cost, the shorter time it will take to complete, and the less complicated the process will be for the teams involved in the migration—so it’s important to keep this in mind from the get-go. Ask yourself, “can we archive some of this information instead of migrating it?” and even more importantly, “can we consolidate some of these records?” This can help you to reduce the overall record count and achieve a single source of truth.
1. “Risks and Challenges in Data Migrations and Conversions.” Gartner. 25 Feb 2009.. 16 March 2017. Web.
2. Mistry, Dhrien. “Big Bang vs Phased – the ideal migration approach.” Xceed. 21 Nov 2014. 17 March 2017. Web. < http://www.xceedgroup.com/xceed-blog/big-bang-vs- phased-the-ideal-migration-approach>