Skip to main content

Build a quality framework around your Big Data

Rachel Wheeler Archive
With the influx of information flooding organizations in an era of Big Data, businesses will need to spend less time on the collection and storage of all those details and focus more on how they will structure their databases, avoid data silos and ensure they are able to actually derive knowledge from what they have gathered.

Jim Harris explains in a post for the Data Roundtable that the volume, variety and velocity of Big Data have dominated many conversations about the technology, yet it's also necessary to discuss data management in terms of quality and structure.

"In order to move the big data discussion forward, and, more importantly, enable our organizations to develop strategies for using Big Data to solve business problems, we have to stop fiercely defending our traditional data management perspectives about structure and quality," Harris argues.

To do so, businesses, nonprofits and other organizations will have to be slow and methodical about how they increase the volume of data they use and how they apply it to make operations more efficient. With the vast differences in the kinds of data, companies will need to put them into context - practices for weeding out duplicate customer records will not translate directly to processing social media data and understanding consumer sentiment. While the former class of data will require a "highly structured, qualitative approach," the social data will need a "loosely structured, quantitative method."

Applying his thesis to the task of managing lists of customer contact details, including emails and addresses, taking the time for email validation and ensuring data quality now will save companies the hassle of digging out from under the pile of unqualified, unorganized information later.

Managing Big Data's risks and complications

The importance of having a clear goal and planned path when embracing Big Data is explained in a post on Verisk Analytics' blog, written by Peter Marotta - the firms' enterprise data administrator of enterprise data management - and Virginia Prevosto, the vice president of the Insurance Services Office information services department.

Without the strategy and management map, attempting to analyze the vast amounts of information for more intuitive and efficient marketing, sales, customer service and operational efforts can result in failure.

"Creating better, faster, more robust means of accessing and analyzing large data sets can lead to disaster if your data management and data quality processes don't keep pace," the co-authors write.