The Idaho Transportation Department reduces duplicate records and takes a proactive approach to data quality.
About the Idaho Transportation Department
The Idaho Transportation Department is a government organization responsible for maintaining the state’s transportation infrastructure. Its five branches include aeronautics, highways, motor vehicles (DMV), human resources, and administration. Located in Boise, Idaho, the department works to provide the state’s 1.6 million residents with safe, efficient mobility as well as economic opportunities through improved connectivity.
Providing top-tier services to residents is at the heart of the Idaho Transportation Department’s mission. To meet their goal, the department needs to have a consolidated view of its residents. But information that is spread across siloed databases prevents the department from accurately matching citizen records and, consequently, from providing the best experience possible for the residents of Idaho.
The agency has a legislative mandate to produce one record per DMV customer. In the past, they ran into situations in which they couldn’t identify repeat customers when they came in for different services. This was due, in part, to the many separate databases the department maintained. For instance, driver services had their own client database, vehicle services had their client database, and motor carrier services had their client database.
According to Randi Bristol-Hogue, DMV Transformation Program Manager, “Until we got to a position where we could bring that data together and identify where those clients were the same, we couldn’t really provide the level of service that we wanted to.”
To further complicate matters, the department’s mainframe data table structure is non-relational, so every record in the mainframe represents one transaction. This means that if a resident owns three vehicles, they will have three records even though they are the same person. In a DMV operation, there is a large diversity of datasets, including driver’s licenses, motor carrier registrations, domestic Idaho carrier records, and foreign carrier records for those who only pay for miles they drive through Idaho. This translates to a lot of information that needs to be tied together.
In 2010, help came in the form of a mandate by the state legislature. In an effort to consolidate duplicate records for individuals, the Transportation Department lobbied for the state of Idaho to put legislation in place that requires residents to provide unique identifying information (such as a Social Security number or driver’s license number) to the department. According to Bristol-Hogue, “The proposed legislation requires residents to give us enough information so that we can adequately match them if they already exist in our system somewhere.” With the mandate in place, the transportation department had what they needed to match customers to their records.
While the mandate was a boon for achieving their goal of having one record per DMV customer, it also created unique challenges for the department. The influx of data highlighted the need for more robust data management capabilities than their existing mainframe environment could provide. Any new system would need to provide long-term storage of the data for historical purposes, enable reporting, and have an ongoing process for deduplicating the data.
The Idaho Transportation Department wasn’t having success with their current solution, both in terms of its performance (the ability to meet deduplication and quality standards), as well as its operational demands. For instance, their old solution required frequent dataset updates that would take the system offline, causing issues across the department. With no other option in sight, they resigned to trying to make it work.
In 2016, they started conversations and explored what Experian Data Quality had to offer. Initially, the department was interested in solutions for address verification, as well as a solution that had matching capabilities. While these solutions take care of the performance issues the department was experiencing, they still needed to solve for the operational demands that limit their current solution. After a seeing a demo of Experian Pandora, the department felt they might have found their solution. “I think that’s when the light went on that maybe there is something better that’s worth the effort to make the switch,” says Bristol-Hogue.
Putting forward a case to implement the Experian solutions was straightforward for the department. According to Neil Snyder, DMV Software Architect, “When the possibilities of the Experian products came along, the justification was really an economic one. We could swap out the existing product with the Experian product—get more bang for the buck—and from a cost perspective remain neutral.”
Significant reduction in the number of records
The Idaho Transportation Department is well on its way to achieving its goal of having one record per DMV customer. According to Snyder, the department started out with 20 million independent records in their dataset. Given that Idaho only has a population of 1.6 million people, this provides some sense of the magnitude of duplicate records in their database. Through previous efforts, the department was able to go from 20 million to 8 million records. Their goal with Experian Data Quality is to have fewer than 3 million records, and they’re confident they’ll achieve this goal.
Using Experian Pandora, the department implemented a rule change that will make it easier to deduplicate customer records. Historically, the department required three identifying matches (name, Social Security number, and driver’s license number) in order to mark a record as a duplicate. With Experian Pandora, they are able to do an analysis to see where they have Social Security numbers and where they don’t have Social Security numbers, but have a driver’s license number instead. According to Carin Pluto, Senior Database Administrator, “With Experian Pandora, we were able to get approval to deduplicate some of these customers based on two matches, rather than all three. While we could do the same thing though SQL queries, it was much easier to visualize using Experian Pandora.”
Improved efficiency and transparency
The benefits of Experian Data Quality’s solutions extends far beyond the goal of reducing duplicate records. The department has become much more proactive in its approach to data quality and has really empowered its developers to use their data assets more efficiently. “There’s often a disconnect between the people with the skills to write the SQL queries and the people who know how to look at the data. I see Experian Pandora providing a direct path for the people who understand the data to actually get in and extract the output, rather than having that intermediate layer of a SQL expert who has to comprehend what the business goal is, and then turn it into the proper technical queries to produce the output,” says Snyder.
Experian Pandora’s built-in visualization capabilities help the department communicate data issues with more clarity than they previously could. According to Pluto, “The developers working with the data were aware of the issues, but Experian Pandora gave us a way to highlight a bunch of different problems that can be easily displayed or made visible to others.” When the developers spoke about the issues, it didn’t really resonate as much as when they finally saw the dashboards in Experian Pandora with rules defined against them.
In addition, the developers are using Experian Pandora to share these dashboards regularly with executives and managers. This level of transparency keeps stakeholders at all levels interested and engaged with the program, and feedback from management has been positive.
Implemented proactive policies and rules
Experian Pandora also makes it easy to bring recommendations forward to the department’s data governance council for consideration. Snyder says, “We found it valuable to use Experian Pandora to better convey the current status of a dataset and to predict what the results of a rule change may be, and to do it in a more visual, easily consumed way.”
For example, one of the changes they proposed was to standardize suffixes. Often, there are multiple ways suffix information can be written—2, 2nd, II, Jr., etc.—and this creates inconsistencies in the data. With Experian Pandora, they analyzed the distribution of all the various suffixes in the data and recommended a consistent method by which to standardize on those. “Being able to visually display the Experian Pandora output made it easier,” Snyder adds.
The department is using Experian Pandora to implement proactive measures to ensure consistency moving forward. According to Pluto, “We used Pandora to identify issues in our data, and we have built in a standard dropdown in our new system for making sure it stays consistent.” Experian Pandora helped not only to identify the issue, but also to implement preventative measures for keeping it in the new system. In addition, they have implemented SQL code that will take the various patterns and standardize them as they load this data into the new system. According to Snyder, “We’re using the approval to clean up our old data as well as build the new system.”
With Experian Data Quality’s solutions in place, the Idaho Transportation Department is well on its way to achieving one record per DMV customer. More importantly, they’re finally able to provide more efficient, high-quality services to their constituents. The changes they have implemented in their new system will help to ensure that the quality of their data is maintained over the long haul, bringing them one step closer to being the best transportation department in the nation.