Looking past the big data hype

Technology vendors may have jumped on the bandwagon by encouraging organisations to build ‘big data' repositories that track every aspect of their business, but that doesn't mean everyday organisations are even close to figuring out how to deal with it.

Indeed, many are so caught up in the mechanics of big data – how to store and manage dizzying volumes of data about nearly every aspect of the company's operation – that they're still struggling to elucidate a big-picture big-data strategy. Executives may love the promise of new business insights, but when it comes time to deliver the reality is something else entirely.

Marco Schultheis knows all too well about the demand side of the equation: big data and better analytics have become matters of survival within the university sector. And as executive director of strategy and planning with Perth-based Curtin University of Technology, he's front and centre in a viciously competitive sector that is now guzzling student-enrolment data to better model behaviours and marketing effectiveness.

“Over the last few years the reliance on data and the appetite for decision-making have grown,” Schultheis told attendees at a recent Big Data Analytics conference in Melbourne. “As the market becomes more competitive, people are less inclined to make decisions based on intuition; they want to be more agile and responsive, and want more data to base their decisions on.”

The promise, the pain

More data, however, means nothing but heartache unless companies can develop appropriate strategies for using that data. This means the gap between data collection and decision making has never been bigger, and it's going to get more so as companies default to storing massive volumes of data that they may not even have a use for.

IT executives need to make sure they don't get pressured into over-promising and under-delivering by managers who have eaten up the hype. All the usual rules for data management, which have been well learned by many early data-warehousing advocates, apply: data cleansing, for example, is an absolute must because dirty data is only useful if it's consistent and usable.

“If the data quality isn't there, everything else you do is garbage,” warned Paul Ormonde-James, director for business solutions and insights with hospitality and convention giant Echo Entertainment Group, which is exploring big-data options to better model the behaviour of casino patrons as they consume various venue services.

“That was one of the big challenges we had: we had aspirations of building all this wonderful forecasting, but when you calculate the errors, the errors were bigger than the things you were trying to forecast.”

His solution has been to focus on small wins first – identifying a “real, tangible problem that I think people want an answer to” and getting their strong support to put some early, high-profile wins on the board. He uses weekly review cycles to ensure regular updates and avoid project disasters, and to win the support of executives that are “really shy about risking money” on massive big-data exercises.

“I try to build initiative that show something,” he said. “If you can directly impact revenue growth, you can build that relationship.”

When it's done right, big data can certainly deliver rewards, especially for companies working to model the behaviour of large numbers of customers. For example, energy companies are capitalising on the introduction of household smart meters and improved distribution-network infrastructure to build vast repositories of minute details that, taken together, can paint startlingly accurate pictures of customer behaviour.

This, in turn, can drive more targeted advertising and product development than ever – as long as companies can figure out what to do with the data.

“When we set up a metadata warehouse, we had the opportunity to work with the different decision makers,” said Daniel Collins, who served as principal business analyst with a “large energy distribution company” and worked to put dollars on every part of the distribution process.

“Six or seven billion meter readings was a starting point,” Collins said. “But the question was: what do you do with it? There was previously a feeling that teams really weren't understanding the limitations they were operating under: when they went out and collected bottom-up data and incorporated that into commercial simulations, it has really allowed questioning some of the assumptions about how they do things.”

Slow beginnings

Despite the hype around big data, however, many Australian organisations are still proving slow to get up to speed with their strategy.

Some struggle to relate the idea of ‘big data' – often demonstrated by multinational vendors that talk in awestruck tones about companies with petabytes of online data – and fail to see how it relates to their more-modest data environments.

“Australia has been a bit slower to come around to the potential value of big data and the problems associated with it,” warns James Foster, practice lead for high performance analytics with analytics giant SAS.

“Customers in bigger markets recognise that there needs to be multiple data strategies; they're implementing analytics data markets, enterprise data warehouses – maybe six to 12 of them. In Australia we've got our official enterprise data warehouse and ‘this other thing' we're building'. The challenge really is around defining that ‘that other thing' is.”

Analyst firm IDC recently latched onto these complexities, noting that a looming explosion in collected company data, confusion over the term ‘big data' has created “unprecedented levels of complexity for IT executives.”

“As the variety of data sources increases rapidly and the velocity at which data is generated also increases,” IDC Asia/Pacific Business Analytics Research director Craig Stires said, “IT executives are beginning to realise that these massive data sets cannot be processed, managed, and analysed using traditional databases and architecture.”

To keep up, Stires says, IT executives are “looking at a major reassessment of both infrastructure and information in the face of heightened expectation from line of business executives.”

In other words, IT executives bragging about the potential of the big data revolution are finally going to have to put up or shut up. Executives know the value of better information, but the big data flood is coming so thick and fast that nothing short of a complete architectural redesign will let them keep up.

Big data, big challenges

In the real world, as is so often the case, execution is not quite so simple. The merging of structured and unstructured data has challenged data-management vendors for years, and despite their analytical promise most big-data repositories will face the same integration challenges they've been wrestling with for years.

Paired with the pressure of managerial expectations, many companies face the same confluence of challenges that accompanied the initial creation of data warehouses a decade ago. “Data warehouses were meant to be, and are, the solution for that problem,” said Schultheis, “but the challenge we've had with our executive is actually showing outcomes.”

“Our business-intelligence developments always tend to start and then run forever – but we're now working to actually deliver some quick wins right up front, to show we're delivering something in a reasonable timeframe.”

Collins has seen similar issues, with big data efforts winning friends faster through small wins and small chunks of data, rather than getting bogged down as executives try to do too much with too big a chunk of big data.

“The approach is really to do things at a low cost initially, and make do with whatever we can,” he said. This takes the risk out of getting those first few bits of value. We can then say ‘if we invest more in it, we can have this capability to give you the right answer'.”

Another challenge for big picture big data advocates lies in its sheer scope, and the pressures of building a data-driven architecture and management framework.

Such efforts have been tried before with varying degrees of success, Foster warned, and just because someone is excited about the idea of big data doesn't automatically mean it's going to work this time around.

“We've got organisations saying ‘we really want to hear what our customers are saying and want to tap into LinkedIn for big data,” he laughed. “These are the same organisations that have been sitting on ten years of customer feedback forms that are buried in a drawer that nobody has ever really analysed.”