You may have heard that Big Data is the new Rock ‘n’ Roll in the world of analytics. It’s been all over the internet and the traditional press for the past couple of years. From finance to pharma, supermarkets to the science sector, big data is going to be the means that takes business from bubbling under to going straight in at number one with a bullet and getting that elusive platinum disc.
Thanks to the constant innovation in technology, the collection and storage of vast amounts customer information has never been easier than it is right now and it’s only going to get quicker and simpler. The world of tools we analysts now have to play with to make sense of this info is growing in tandem at a rapid pace too.
There are now mountains and mountains of new data being produced everyday from a variety of interesting, worthy and valuable sources. Alongside this there’s the avalanche of shiny digital toys with which to slice, dice and interpret all this information. This should make all us analysts very happy you would think. And we are, sort of. But there is still an outstanding issue that every analyst faces – data quality.
Living by numbers
There have always been problems with data integrity. But now there’s just a whole lot more data to clean up before we can get to the insight. The 2015 Experian Data Quality benchmark suggests that, on average, US companies believe 32% of their data is inaccurate. That’s right, there’s just shy of a third of all data produced which we should be approaching with caution.
This poor data impacts how effective we can be as analysts. We have to spend more time ‘cleaning’, making approximations or just simply ruling out data sources. If customer data is captured inconsistently or incorrectly, analytical insights can be misleading. In auto-adapting models, the risk is even more significant as the computer brain, unless instructed to do so, may not see how unreliable the captured information is and could direct customers down the wrong contact strategies. It comes back to the age old saying, “Garbage in = Garbage Out”.
So where is it going wrong? There are some data integrity issues that are too big to fix, inherent from legacy systems. We’ve all encountered the aging, patched up systems that have had work around after work around just added to make sure business as usual can continue. Too few have been circled back on and fixed properly. These systems require large scale overhauls, and major investment in data governance. Fixing them is going to be expensive, time consuming and will need buy in from the guys at the top or it’s just never going to happen.
But there are areas where we can improve and influence change. For starters we can advocate, enthuse and generally shout about the third party solutions that are out there. Ready made packages that can go over the top of those unwieldy existing legacy systems. They cost a fraction of the price in both time and money compared to fixing what is there or building from scratch and will eliminate that errant one third of the data. We should also be looking to make improvements in the processes where data is captured manually, like in most call centres. As analysts we should have the courage of our convictions, step up and champion the promotion of better, cleaner and more reliable data whenever we can.
We built this city
It often feels like we analysts are the only ones who see just how much potential value is lost from poor and unusable data. It is frustrating when we know how it could benefit businesses in terms of targeting the people we should be talking to, at what price and when we should be doing it. We need to start talking to many more departments in our organisations to emphasise the real value of data capture. We need everybody that touches the data, directly or indirectly, to realise that it is a monetizable asset and that it’s worth is devalued if it’s not captured correctly.
But it’s not about lecturing people. We need to take people on a journey with us, show how they can help us understand and improve the information we capture, get them involved in shaping the analytics to turn data into insight that benefits the whole business and our customers.
The upside of all of this is that with better, more reliable data, we analysts can spend more time building new, exciting and most importantly, meaningful models for our customers and bosses. In short we all win!
So next time I don my puff ball skirt and head towards the dance floor for that Starship classic and funky floor filler, when we build this Big Data city, we just need to check our foundations and not be afraid to spend more time fixing them first.
