Technology — 21 November 2014

In this white paper, we draw on forecasts from IDC, Gartner, Forbes, and other publications to show what changes in technology and new business opportunities will come along in 2015 because of changes in the Big Data market.  Companies need this forecast information to make staffing, investment, and IT project funding decisions today and do long-term planning.

We first give a definition of Big Data, since the concept of Big Data is still evolving as its standards and definition are not yet agreed. Then we look at how companies have been using Big Data in 2014.  We describe what changes are occurring in technology and what new businesses will come online in 2015 because of the shifting Big Data landscape and the opportunities that brings.  We provide projections of the increase in sales and well as describe how Big Data and Analytics might impact the IT organization head count and structure.

Big Data = Lots of Data and New Kinds of Data

Big Data is growing in volume to many thousands of TB, creating opportunities for IT companies and giving business another tool to make strategic decisions.

Big Data is three things: (1) unstructured data, (2) new sources of data, and (3) Analytics used to sort through and make use of this data. When one talks about Big Data they usually mean the Hadoop distributed file system and Analytics software.

Yahoo! developed Hadoop to deal with the enormous volume of data that powered their search engine.  Then they gave Hadoop away as open source software.  Hadoop and the tools that sprung that from it—created by Google, Facebook, and others—lets companies process enormous quantities of data that would otherwise not easily fit into the rigorous schema of a relation database, like Oracle.  So Big Data makes possible Analytics that was not possible before.

Beyond the development of Hadoop and programming tools like Pig, what shoved Big Data ahead in the market and in people’s thinking was cloud computing. This greatly reduced the cost of expanded computing power. Now companies can deploy hundreds or thousands of data nodes on cloud virtual machines without an upfront capital investment.  Hadoop requires hundreds or thousands of these machines, because it stores and processes data across many computers (the Hadoop Distributed File System) in parallel fashion.

To help measure the market, Wikibon defines Big Data as the following:

  • Hadoop database, software, hardware, and services
  • In-memory flash databases
  • Business intelligence and visualization platforms
  • Cloud-based Big Data infrastructure
  • Advanced analytics and data science platforms, tools, and services
  • Next-generation data warehouse and analytics database software, hardware, and services
  • NoSQL database, software, hardware, and services
  • Application development platforms


Big Data Facilitates Analytics

What powers Big Data is Analytics, also written with a capital “A,” showing that that concept too has not yet matured into a common noun.  Analytics is the set of mathematical and statistic tools by which one can draw conclusions, make inferences, forecasts trends, and finds correlations between different data points.  It is valuable to business, because it lets business harness the power of advanced statistics coming from multiple data sources, without having to employ an army of statisticians and data scientists and without requiring programmers to convert all of these different data formats into a common format. Plus it allows processing real-time information coming from social media and sensors.

With Big Data and Analytics, business can capture data from different sources to make tactical and strategic decisions.  This replaces guesses, hunches, experience, and empirical observations with hard science.

With these tools, businesses and other organizations can decide, for example:

  • Should we open a new store?
  • How should we set our prices to maximize profit?
  • Should we acquire company XYZ?
  • Should we decrease insurance premiums on patients who exercise?
  • Is energy consumption increasing in buildings in certain areas?
  • Is our company’s wellness programs working?
  • How much does a new employee contribute to income and should we add more?
  • Does a college education add value to an employee’s performance?
  • What opportunities are there in emerging markets due to changing events on the ground?

Personal Data = New Sources of Revenue = Privacy Concerns Too

The science of Analytics has a dark side when it comes to privacy issues.  So far there are no regulations with regards to that, at least in the USA.  But the European Union has passed privacy laws that when approved by the European Commission will definitely impact the mining of social networks and web surfing habits via browser cookies. All of that is an important component of Big Data, at least with regards to measuring consumer behavior.

Retail companies have found that information on the buying habits of their customers is valuable information that can be sold.  The CBS 60 Minutes television program explained that it is now possible to use Big Data Analytics to match up social media contacts, website logs, GPS tracking information, and financial information from Experian and others to determine someone’s email address, name, sex, age, physical address, and telephone number.  Data brokers dump all of this data into Hadoop and NoSQL databases and then run Analytics against that to provide customer-specific market information, which facilitates targeting advertising to a single individual.

For example, for targeted advertising online, it is now possible to determine who fishes, who is shopping for a new car, and who is pregnant. The future where computers track our every movement is already upon us.  Scenes like in the Tom Cruise movie “Minority Report,” where electronic signs pitch personal ads as a person walks through a department store, are already possible because people’s cell phone broadcasts their IMEI or serial number, thus identifying the person.

In this paper we do not report on Minority-Report products.  But as anyone who uses Google Chrome knows, such data tracking is already a reality.

The Market in 2014 and the Current Situation

This section explains the technology used and the revenue generated in the Big Data market as well as how some companies are using the technology.

Forbes says, “From manufacturers looking to gain greater insights into streamlining production, reducing time-to-market and increasing product quality to financial services firms seeking to upsell clients, analytics is now essential for any business looking to stay competitive.”

This has definitely translated into increased opportunities for vendors in the Big Data market.  Wikibon says the market for Big Data hardware, software, and professional services was $30 billion in 2014. Sales were $18.6 in 2013, which was a 58% increase over the previous year. Projections for 2015, which we explain in a moment, show the market growing at least 30% per year for the next 5 years.

How Companies are Using Big Data

How did organizations use Big Data in 2014?  What key challenges did they face?

Tech Republic says that one challenge customers face when trying to process Big Data is that vendors have not yet agreed on standards.  This means there is no common tool and no common approach.

Another problem with dumping all available data into the Hadoop cluster and then trying to make sense of it is what people in communications would call the signal-to-noise problem.  That means separating data that is of no value (noise) from useful data (signal).  Examples of that are, say, a webserver and router log that includes handshakes and not just web pages that people actually opened or servers they actually visited. It takes time, effort, and tools to eliminate that clutter.

One solution to the proliferation of tools and lack of standards is the development of APIs to allow programmers to access the same data repository using different programming languages.  This is what Apache Spark does.  The CEO of Databricks told Tech Republic, “”When you use Hadoop, Storm, GraphLog, and other big data solutions, each comes with its own set of tools. Each also uses its own programming language, whether it is Java, C++, or something else. For enterprises constructing their own big data pipelines, it is left up to them to stick these tools together into a collective infrastructure that can manage all of the various areas of the pipeline.” He said, “”With Spark, a site has to worry about managing only one API (application programming interface), which makes managing the data pipeline easier.”

Since it has the name “Apache,” in front of it, that means the developer are giving Spark away for free as open source software.  Databricks and others then built products on top of that or individual companies are free to download it and use it themselves.

Using Big Data to Make Pricing Decisions

Here is an example of how companies have but the Big Data ideas into practice.

McKinsey comes close to mocking the traditional way that companies set the prices of their product, accusing them of basically guessing or using the old-fashion practice of just basing it on cost.  They say, “Many marketers end up simply burying their heads in the sand.” A better approach they say is to use Big Data Analytics to set prices based upon the optimal price point customers will pay.

McKinsey cites different techniques of Analytics to do that. On is dynamic deal scoring, which means looking at individual deals.  Also Big Data and Analytics lets a company study individual receipts instead of aggregate sales. For example, a company could look at how much of product X someone bought given that they bought n units of product Y and product Y was on sale.

The Outlook for 2015

ICS, Gartner, and others have interviewed IT executives about their buying plans for 2015.  Gartner, for example, surveyed 2,800 CIOs in 84 countries.

Their research gives us some predictions. Here are three:

  • ICS says that the market for Advanced and Predictive Analytics software is expected to grow from $2.2 billion in 2013 to $3.4 billion in 2018. The top 3 vendors are SAS, IBM, and Microsoft.
  • T. Kearny says spending on Big Data hardware, software, and services will grow 30% per year from 2014 through 2018 reaching $114 billion.
  • Cloud-based business intelligence is expected to grow 31% per year through 2018 reaching $2.94 billion.

Not all Businesses use Big Data Yet

Tech Republic points to the IPO of the Hadoop vendor Hortonworks as lending more legitimacy to the Big Data market. Not all IT executives have added Big Data to their matrix of systems as their traditional transactional systems and spreadsheets are all they need to run their businesses, they say.  (To say that Hortonworks is a Hadoop vendor means their version of the open source Hadoop software comes with support.)

They write, “If there is a real demand for big data and Hadoop, an IPO by Hortonworks could help further legitimize the big data space and grab the attention of apprehensive corporate IT decision-makers who have avoided big data so far.”

The web site points out, “It’s important to remember that Hortonworks still isn’t a profitable company. In fact, its losses are growing. Its revenues are steadily growing as well, but they still aren’t on par with the amount of money the company is losing.”

Changing Technology

Hadoop is a batch system.  That makes it hard to obtain data in real time, such as streaming data from social media.  If a particular item is trending on Twitter, and a company cannot match that up with their product mix, then this could mean missed opportunities.

Flash memory has plunged in prices allowing in-memory databases and reducing seek time on storage from microseconds to nanoseconds.  That helps data Hadoop MapReduce jobs run faster.  But it still does not provide real-time SQL-like access to data.  Facebook developed Hive to provide near real-time access to Hadoop, but it is not quite there yet.

When Netflix talks about streaming they mean streaming video, but when Big Data vendors talk about streaming they mean processing Twitter feeds or any kind of data as it comes in real time, like sensor data or stock prices.  Cloudera has announced new initiatives to further promote Apache Spark as the tool of choice for process streams.

Here are some specific ideas for Big Data products that entrepreneurs have turned into businesses this year:

  • Scalable Inference has created tools for cloud-based machine learning.
  • GridCraft has created a web-based spreadsheet-like interface to work with analytics. This is a new way to look at the data instead of visualization, meaning graphs that let you drill down into the details.
  • EQLIM has turned measuring strife and turmoil in the developing work into a business. They have created a subscription service “for real-time data about human activity in emerging economies.” This includes metrics like energy consumption, commerce, agriculture, and buildings going up and getting town down or blown up. Hassan Alassaad, CEO and cofounder, says they are “basically mapping in real-time risk and opportunities across the region.” He says this is a billion dollar business.

IT Staffing

How will the adoption of Big Data result in changing traditional roles within IT in an organization? For example, will companies appoint a Chief Data Officer?

Gartner says, “75% of IT executives say that they need to change their leadership style in the next three years.” They say that by 2018, companies will need 50% fewer process-oriented staff and 5 times as many “digital business jobs.”  They classify digital business jobs as:

  • Integration Specialists
  • Digital Business Architects
  • Regulatory Analysts
  • Risk Professionals

Another report says that in 2014 and 2015, the number of Chief Data Officers and Chief Data officers has or will double.

New Businesses created by Big Data

Change is always good for business, as it creates new opportunities, as long as business can harness that. But there is a downside to having all information online as that has made competition tougher. Forbes says, “Pricing and customer satisfaction information has eroded the competitive advantage afforded in the past by product innovation and is shifting attention to customer experience innovation as the key to a lasting brand loyalty.”

This proliferation is about to get a whole lot more prolific. “The Internet of Things” means that many devices that were offline before have been or will be connected to the internet. This includes things like automobiles, people (health monitors), machines of all types, and sensors.

Some of the businesses that have been or will be created by the advent of the Internet of Things and the ability of Big Data and Analytics to sort through all this information include:

  • Wireless health monitoring will create opportunities for doctors and patients.
  • Mobile digital assistants, like Siri on the iPad and Google’s voice recognition, will help customers fill out data on web sites and accomplish other tasks.
  • Tracking your children and their progress at school relative to others.
  • The use of mobile payments will increase, driven by Apple Pay and the increased hacking of less-secure credit card systems. This will breed new life into the stalled efforts of payment systems like the Google Wallet.
  • There will be more 3D printing of consumer goods, which is not directly Big Data related, but is information-related and will benefit from Big Data targeted advertising and new technologies created for the Big Data platform.


Big Data is big business.  Some companies are still trying to figure out what it means, while software vendors and consultants are working hard to educate them.  Those who have already climbed the knowledge curve have found new opportunities in mining Big Data and Analytics to either create new businesses or improve time-to-market and profits in existing businesses.

But one item missing from these forecasts is much of what is changing in the market is not going to affect business in the developing world, which is where most of the world’s people live, for some time yet.  The USA will take the lead on new developments of technology with Europe right behind them.  Uber, for example, is a perfect example of an idea conceived in the USA but roiling European markets now.  But Latin America, the Middle East, and other economies are going to take some years to catch up to the Big Data phenomenon.  That in itself is a reason to look for new business opportunities in the developing world as new products developed in the developed world reach maturity.


Shacklett, Mary. “Big data trends in 2015 reflect strategic and operational goals” Tech Republic.” 3 October 2014.

Columbus, Louis. “Roundup of Analytics, Big Data & Business Intelligence Forecasts and Market Estimates, 2014” Forbes. 24 June 2014.








About Author

(0) Readers Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

8 − three =