Big Data….Big Deal?

There is a lot of hype in the world of reporting and analytics at the moment around Big Data. Is

Big Data the silver bullet for all BI problems or is it just a passing fad?

Traditional definitions of Big Data refer to it in the context of Volume, Variety and Velocity. In this

sense, it is about more than just size, it also about different types of information such as structured

and unstructured, and the speed with which this is generated and changes. For example, retail giant

Wal-Mart feeds more than 1million transactions an hour into databases estimated at more than 2.5

petabytes. Facebook’s estimated 750 million users create an average of 90 pieces of content each

month and an average of 294 billion e-mails are sent every day.

Despite this, Big Data is in some ways not a new concept. Organisations have always been adept at

generating large amounts of data and the explosion in variety and velocity through things such as

social media and the incessant chatter of the ” internet of things” threatens to be the proverbial straw

that breaks the camel’s back.

The real challenge with Big Data lies in creating meaning through insight and it is here that

technology enablement is at an inflection point. Advancements in analytical capabilities are now at a

point where the creation of meaning from mass is achievable. Economist Intelligence Unit research

indicates however, that most businesses have made slow progress in extracting value from big data

and some companies attempt to use traditional data management practices on big data, only to learn

that the old rules no longer apply.

Technology is however only part of the solution and having a technology centric approach will likely

result in organisations being overwhelmed by information irrelevance. Data does not make

decisions, people do, and more data does not automatically result in better decisions.

Miller’s Law suggests that the number of objects an average person can hold in working memory at a

point in time is 7 plus or minus 2. While the ability of technology to store, process and sort data has

grown exponentially over time, human cognitive ability has arguably not grown at the same rate. The

challenge for Big Data is therefore one of relevance – how do you synthesize the masses of data into

the 7 plus or minus 2 objects to support effective decision making?

Keeping this in mind Big Data does offer tremendous opportunity to drive value. A recent study by

McKinsey has suggested US retailers could realise in excess of a 60% improvement in net margin and

consumers globally 0 billion in value through the creative and innovative use of Big Data. They have

further indicated that Big Data will become a key basis for competition and growth.

While another global survey In June 2011 by the Economist Intelligence Unit of 586 senior

executives, looked at the state of big data, the organisational characteristics of companies that are

adept at extracting value from data and the most challenging aspects of data management. The

research found:

Extracting value from big data remains elusive for many organisations. For most companies today, data is abundant and readily available, but not well used. Nearly one in four said the vast majority of its company’s data is untapped. Another 53% say they only use about half of their valuable data. Yet 73% say that data collection in their organisation has increased over the last year. These figures indicate that organisations are still learning how to manage Big Data.

Many companies struggle with the most basic aspects of data management, such as cleaning, verifying or reconciling data across the organisation. Nearly one-third of respondents admit their data governance practices are insufficient. Many struggle to deliver important data to the right people within an acceptable timeframe. And there is also a depth of workforce skills required to sift through, analyse and develop insights from Big Data. The lack of the right skills to manage data effectively is among the top two challenges cited by CIO’s. While the most difficult process right now is reconciling disparate data sources.

Companies that are furthest along the data management competency continuum provide a useful model for how organisations will need to evolve if they are to extract and utilise valuable data-driven insights. Strategic data managers use data to first identify specific measurements and data points that align closely with corporate strategic goals. They select the most appropriate data to make decisions, and put a high percentage of the data they collect to use. They are also more likely to assign a C-level executive to manage data strategy, and they continue to explore emerging sources of data for potential value.

Of course, Big Data will be but one of the tools that companies use to inform decisions. But

companies if they are going to effectively harness Big Data need to start with the end in mind and

have a razor-sharp focus on outcomes. It also requires some unique people skills in terms of data

management and the ability to harness data as a critical factor of production on a par with land,

labour and capital.

To understand how we tackle the issue of Big Data in the world of Microsoft go to the Inside Info

website by clicking here and download the Big Data Info Sheet.

Telephone: 1300 768 110

Email: info@insideinfo.com.au

Head Office: Level 8, 75 Miller St, North Sydney, NSW 2060 Australia