How important is big data?

If you want proof that big data is a big thing, just look at the jobs pages. Inside, you'll find desperate appeals for statistics-savvy data scientists, and the salaries are often eye-watering.

They're evidence that businesses in the UK and beyond are looking at ways of using the data they already collect – whether that's about customers, their industry or about financial markets – with as much efficiency as possible.

Digital markers

Data is everywhere, it's doubling every two years, and it's fast becoming a business' biggest resource. The digital markers we all create – whether a GPS-tagged photo, check-in on Facebook or a click on a website – could last forever.

So far, we've got about 1.8 zettabytes of data on planet Earth, but it's how we link up, analyse and use all of this data – and much more besides – that's going to completely change our world. And that means business.

Some say that by mastering big data, companies can stop wasting time on useless initiatives and pitches that go nowhere - and increase margins by a whopping 60 per cent.

Even if it's a fraction of that figure, harnessing big data properly can make the difference between a profitable business and a non-starter.

Big data is big money. ABI Research states that global spending on big data by organisations exceeded US$31 billion in 2013, and will reach a staggering US$114 billion in 2018 – and that includes salaries as well as hardware and software.

It's a data-rich, skills-poor environment at present, but the future will be one of increasingly sophisticated hardware and ever-better data analytics that collect, calculate, predict and much more besides.

The iceberg problem

Unfortunately, big data is scattered. It's estimated that IT departments spend about 70 to 80 per cent of their time maintaining existing systems and databases, largely because there are so many of them.

Separate, created for narrow purposes, and often based on databases that don't talk to each other, it's what's known as 'the iceberg problem', where systems can't exchange big data, which therefore goes untapped.

"One of the companies we're working with has 6,670 'icebergs' used by various arms of the business," says Dr Andrew Sutherland, Senior Vice President of Technology at Oracle EMEA.

"Some are very important, each one was built for a reason, and all had and still have a business case; and they were all built by very clever, experienced people – but each application has been built to fulfil one need, and they use completely different databases."

False economy

It's a common problem, but businesses who've invested a lot in such icebergs often don't want to replace them. For a business wanting to embrace big data, that viewpoint represents a false economy.

Petabytes of big data are now flooding in to businesses and a consistent, common platform is required, capable of exchanging data and running apps – whatever their source – using only one stack that can be managed, maintained, recovered and backed up as one.

It's all about consolidation. "Whether you're a police force, a gas extractor or a retailer, there will be areas of your business that you want to be able to do more efficiently or cleverly – and you can only do that if you consolidate and simplify your IT, so you can concentrate on the apps that add value to your business," says Sutherland.

One problem here is that there often exist huge teams of IT consultants whose job it is to specify and commission more icebergs; asking them to give their blessing to using one unified, interoperable system that best takes advantage of big data is like asking turkeys to vote for Christmas.

But if you do get big data into one place, what happens to it next?

Machine learning & super-computers

The big data revolution will be led by machine learning and super-computers. They've been called the 'coal in the furnace' of the internet that will drive Web 3.0, but the sole reason for the new generation of artificially intelligent super-computers is the massive rise in personal and business data.

Eventually enabling websites and apps to track and predict our every action and desire, super-computers feed off big data.

Reasoning, perception, social and even some degree of cultural awareness are the selling points for Watson, IBM's super-computer that runs on 16 terabytes of RAM, and brings a much-needed injection of artificial intelligence to big data analytics.

Its smart learning DeepQA software is able to search databases, spot patterns and make predictions, which could revolutionise the financial and medical industries, both of which are currently drowning in data.

Big data in medicine

As well as reducing the need for investment bankers, big data analytics could also banish bad decisions by doctors.

"Watson could be used to search through millions of pages of academic research and drug trials, something a human could never do or keep up with, and in double-quick time," says Joe Peppard, a professor at the European School of Management and Technology in Berlin, Germany, which consults on IT strategy for large corporations.

"It could provide a doctor or nurse with the required diagnosis and even the most appropriate medication plan, having taken into consideration all the latest thinking on any specific health problem."

Call centres streamlined

Such data analysis could also be used to vastly improve call centres, where about 60 per cent of enquiries end in frustration for callers.

"Watson is able to search a far greater amount of data in a much faster time," says Peppard, "so in many cases it could pre-empt enquiries so that the answer is available before the question has been asked."

"Machine learning, and its application in advanced analytics, is one area that will make both the public and private sectors data-savvier than anything we've seen so far," says Dan Shey, Practice Director at ABI Research.

"Big players such as IBM and HP are understandably moving to this direction, but at the same time we can also see analytics startups, like Ayasdi and Skytree, that have machine learning in their very DNA. Eventually, such innovations will put analytics within any domain expert's reach. At that point, data will stop being big."

Uncovering the Universe

Harnessing big data properly can mean new discoveries and possibilities, and nowhere more so than in deep space.

The European Space Agency's Gaia mission to create a complete three-dimensional map of the Milky Way involves taking super hi-res photos of the heavens, but to make any use of this data requires a powerful information system.

Archiving and processing scientific data collected by the Gaia satellite, the Italian National Institute of Astrophysics (INAF) uses a data management system from Oracle to store its "very precious heritage of astronomical data that will have to be stored for the whole 21st century and beyond," says Roberto Morbidelli, Scientific Operation Manager at INAF.

Processing petabytes of big data from Gaia is a huge task, but Oracle has history; it also works with the operator of the Large Hadron Collider, CERN.

The sources and uses of big data in business are now being revealed, but if we know one thing it's that the hyper-efficient collection and analysis of data is already becoming a major tool.

"In five or 10 years' time, big data will not be a source of competitive advantage, it will just be a necessary part of normal business," says Eddie Short, UK and EMEA Leader for Data and Analytics at KPMG Management Consulting. "Without exploiting it, a business will die."