Big Data Analytics: At the Tip of an Iceberg

The Big Question

While humans have applied mathematics and statistical analysis in order to solve problems, for several centuries, the use of analytics to solve critical and complex problems is currently undergoing a technology-enabled revolution.  The underlying driver of this trend is the sheer quantity of data available in enterprise, government and consumer contexts.  According to analysts[2], only 0.5% of this digital universe is currently being analysed, however spending on big data solutions (including software and services) has already reached c.US$10 billion in 2013[3] and is expected to continue to grow at over 30% over the next decade.  However, the impact on productivity across the economy will be significantly larger; because like electricity, the internet and the information applications that have accompanied it, analytics can catalyse the productivity of communities, companies, organisations, government institutions and therefore entire economies.  For example, McKinsey’s 2011 report on big data analytics estimates that the impact of analytics on US healthcare sector could be US$300bn while a US$600bn consumer surplus could be created by using personal location data globally.  While we are still in the very early stages of the analytics revolution, beyond corporate applications, there is substantial potential for analytics applications in personal and social-community decision making and government services.  Today, fully unlocking the value of analytics in complex situations requires both the application of new technologies and computing power; but also significant human intervention to apply judgment to harness the power of these tools to increase efficiency.  While in the near-term, this will result in the development of more offshore centres which combine technology and human processing in trusted, secure, and low cost regions, the longer trend is clearly towards lesser human intervention and more automation.  Amazon, through its automated decision support prompts in interactive systems has demonstrated this to the greatest effect[4].   Even at today’s stage of development, multi-billion dollars of value have been created by corporations based on the use of analytics.  Beyond the significant economic impact, events such as the Arab Spring have been inspired by merely connecting and exchanging information; as information exchange and processing become more sophisticated the possibilities for societal change grow increasingly vast.  As this passes some critical threshold of useability, the information-inspired revolution can now ramp up to another level across the world.  The big questions therefore are how one can invest to take advantage of this revolution, what technologies or human interventions one should leverage, and how and where to prioritise the application of big data analytics in an organisation.

The Potential for Big Data Analytics

The Investor

Big Data Analytics is clearly not just a way to solve business issues but a way to find innovative ways of transforming all parts of society[1].  With this in mind, it is important to view technology-enabled analytics in the context of other critical productivity-enhancing information technologies which transformed the manner in which individuals, businesses and organisations functioned over the last 2-3 decades.  The rapid evolution of information hardware from mainframes to the personal computer to smartphones and the advent of the world wide web, social networks and new software tools have together allowed people to create, store and access information on an exponentially larger scale – in the process fundamentally transforming humans’ productive potential and the way in which social communities can develop and interact.  As a result, total IT spending has reached an estimated US$3.8 trillion (or c.5% of global GDP)[6].  For most of these innovations, American companies have been at the forefront of both developing the technologies and ensuring their widespread commercialisation in the economy – with the real impact of these technologies visible in the sharp increase in US productivity growth from the mid-1990s till date (see chart and quote).

In this context, it is apparent that, led by the US, the world is in the early stages of a new technology cycle which already has and will continue to impact multiple aspects of our lives.  Analytics is now the common factor driving innovation in seven key areas which together describe the outlines of the revolution in our way of life from information:

  1. Personal Choice and Pre-Emption of Choice.  All manner of data is now being used to sell more to the individual and also to enable individuals to manage their life choices.  Personalisation is now an integral part of the sale of any goods and services to the consumer.  The frontier of pre-emption is a huge one that gets closer as analytics help profile and respond real time to events.
  2. Facilitating Social and Community Value Building (and Destruction).  Community-based identities (regional, political, or brand-affiliated or other) are forming online and generating data which can be used to enable easier collective decision-making.  Using analytics, it is increasingly easier to identify like-minded organisations and individuals by leveraging data – which can be applied in something as innocuous as online gaming communities to something as globally significant as political uprisings.
  3. Unlocking the Power of Genetics. Rapid advances in DNA sequencing can be combined with big data analytics techniques allowing for the analysis of whole populations to create targeted healthcare treatments. Having fully decoded the genome, Big Data Analytics now promises to help us unlock the power of the information contained therein.
  4. Corporate DNA Transformation.  Just as previous IT innovations have flattened corporate structures, big data analytics is now being applied at all levels of the corporate hierarchy to transform and re-invent virtually all business processes to make them more efficient and competitive.  Analytics is thus both supporting and replacing human decision-making to help drive efficiency in corporations and other organisations.
  5. Transaction Management.  Big data is transforming how virtually every transaction in conducted – from the sale of books, to music and even corporate M&A.  Analytics allows companies to better target customers and reduce the cost of doing so, thereby putting a tangible value on data assets.  Analytics is teaching companies the value of storing and analysing transaction histories to learn new insights which can help them compete.
  6. Government, Politics, Security.  In an age of prolonged fiscal austerity, leveraging big data has rapidly become a critical link in efficiently managing government services to ensure maximum impact of every bit of spending.  At the same time, the success of political campaigns and the effectiveness of security and intelligence agencies (both cyber and physical) and the prosecution of war depends on the ability to analyse big data.  
  7. Global Cross-Boundary Decision-Making.  Cross-border institutions can now access a wide range of data to address the most complex and intractable issues including climate change, environmental global healthcare delivery, and financial inclusion.  For these issues, and many others which cross-border institutions are trying to address, analytics is rapidly displacing bureaucracy and pontification by allowing governments to develop an empirical rationale for policies rather than relying on ideological justification – thereby challenging built-in ideological biases and leading to a highly contextual policy response.

The Market for Big Data Analytics Solutions

Given that big data analytics has the potential to transform how virtually every decision – from the individual to the global level – is being made, it is clear that the ‘industry’ for analytics solutions, currently focusing on corporate verticals, is in the early stages of a long and transformative development.

The Investor 3There are no consistent estimates for the size of the ‘analytics’ industry as such – primarily because it is still carving a space for itself next to large established sectors including traditional IT services, knowledge process outsourcing, consulting, market research, and data processing while becoming increasingly entwined with the rapid growth of social media, mobility, and cloud computing technologies – thereby blurring the boundaries.  Experts estimate that total spending by companies and public institutions on analytics services and software solutions reached c.US$5 billion in 2012 and expect it to grow at a 30% CAGR over the next decade to reach c.US$50bn by 2020[8] (see chart).  Thus far, the US has led the way and accounts for over 50% of the current demand – with sectors such as financial services, retail, and healthcare being the early adopters.  European companies too are being compelled by the cost-benefit tradeoff of analytics solutions and make up a significant pool of demand.  In comparison, demand for analytics is currently nominal in Asia and other emerging economies.  Therefore the rapid growth of analytics will be driven both by the application of analytics techniques in new verticals and functional areas, as well as by increasing penetration both in new markets – particularly in Asia where large economies such as China and Japan can realise significant productivity benefits from its applications.The landscape of providers of analytics technologies and services that is emerging as a result of this rapid growth consists of several different types of roles to be played by industry participants (recognising that there will certainly be some players who will seek to play multiple roles in parallel):

  1. The “Thinkers”: companies that will focus on developing intellectual property and thereby make significant technological breakthroughs in analytic tools and methods (such as pre-emptive, artificial intelligence and pattern recognition software, DNA analytics, developers of app-bots and avatars, etc.)
  2. The “Arms Dealers”: other companies, particularly the large IT firms will focus on ‘productizing’ the intellectual property developed by the Thinkers and removing the human element and thereby accelerate the process by which these technologies and methods are rapidly commercialised and available in mass.
  3. The “Counselors”: consulting-oriented companies will focus on developing bespoke solutions for large corporates and governments to create strategic insights from analytics tools – and explicitly link analytical tools with the decision-making within these organisations
  4. The “Do-ers”: technology-agnostic service providers who will focus on deploying large armies of outsourced and offshored talent to lay the foundations for deploying analytics tools at scale (including data scientists, coders, data processing and cleaning, database management, etc.)
  5. The “Utilities”: large scale deployment of analytics will require significant investments in infrastructure and allied services – particularly for users who do not have the scale to build captive infrastructure.  Cloud computing and ‘public’ data centres have already made data storage a variable operating expense (rather than a large capital investment); similar models for computing power may also emerge.

The different elements of the analytics landscape above will clearly evolve at a variable rate.  Indeed, some strategies are already reaching a critical mass with large analytics companies with distinct business models starting to emerge.  Creating sustainable value in this rapidly evolving ecosystem requires both service and technology providers to continuously focus on developing strong intellectual property or risk being rendered obsolete by another technology or process.  Within this context, there are a number of emerging high potential areas of focus where analytics can be applied for maximum value creation:

Investor 7

Key Challenges: The Talent Gap and Data Security

Although technological innovation over the last two years has played a critical role in catalysing the growth of analytics spending, the process of conceiving and implementing analytics solutions for a business issue (even if it is software-based) still requires significant human intervention.  The availability of talent is therefore the biggest constraint to the rapid growth of analytics – in the US alone, there is currently a “shortage” of analytics professionals which is expected to increase to 180,000 by 2018[9] based on graduation rates.  Nevertheless, the analytics industry continues to grow rapidly by leveraging skillsets which exist in other industries and functions and through offshoring.  Large firms have built captive analytics units, and pure-play analytics firms started by former professionals from management consulting, the KPO-ITES industry, as well as domain professionals have begun to emerge and build scale.  While the US remains the largest market and currently the leading provider of analytics, the talent shortage has allowed India to emerge as a key offshoring destination for analytics services due to its large, low-cost talent pool and by taking advantage of its leadership position in the KPO industry and drawing on its experience with offshore delivery models to provide analytics services at scale.

Nevertheless, India’s talent pool alone is likely going to be insufficient to meet the rapid demand growth being projected for analytics solutions, which will provide an opportunity for other countries, in particular China, to become key providers of analytics not just in their own domestic markets but globally.  Behind the US, China and India graduate the second and third largest number of people with deep analytical training, and together they graduate c. 24% more analytics experts than the US.  With growing enrolment in higher education, the number of graduates with the relevant skill sets is growing at a rapid pace in these countries relative to the US (China: 10.4% vs. US: 3.9% ) thus making India’s and China’s human resources a critical component in maintaining the growth of the global big data market.  While the talent shortage in the US will provide an opportunity for Indian and Chinese firms to build market leadership in providing big data solutions on a global scale, in order to break out of the ranks of the “Do-ers” and “Utilities”, analytics providers in these markets need to invest in R&D, significantly augment their technological capabilities and build deep domain expertise in order to become integrated solutions providers rather than “body shops” which capture only a small part of the value chain.

Furthermore, in order to become an ‘exporter’ and outsourcing destination for analytics to the West, China in particular will need to overcome concerns around data security which have arisen over the past few years.  Revelations in 2013 about organised government incursions into corporate servers to steal trade secrets combined with the dominance of the government in the network infrastructure[10] will create a fundamental concern around sharing sensitive data.  Of course, as we have noted, the domestic analytics opportunity for China is substantial and could well fully occupy China’s population of mathematicians and data scientists over the short to medium term.  However, to provide these services at scale and in a global fashion, China’s leadership will need to address these concerns.

In the near term, the scarcity of deep analytics talent and the concern around sharing the most sensitive data is likely to drive most companies to develop ‘home-grown’ ways to leverage analytics with minimal human intervention.  It is also likely to drive continued innovation in the technology and algorithms which underpin analytics – driving the long-term move towards artificial intelligence.  Hollywood has often tended to depict AI as something to be afraid of – with imagery of a society gone horribly wrong or machines challenging man.  As recent revelations of domestic snooping by security agencies has revealed, big data will certainly create new philosophical and policy questions about information security.  However, on the whole, the impact of analytics will likely be benign because it provides some measurable value to its users.  As our digital footprint grows exponentially and becomes accessible not just to security agencies to snoop, but also to government agencies who will provide more effective services to citizens and more focused spending of tax dollars.  Companies will gather data, not just to eliminate jobs to improve the bottom line, but also to provide more effective customer offerings, better services to their employees, more efficient supply chains, more energy efficiency, amongst a host of other more beneficial outcomes.

Looking Ahead: Structuring Relationships

While the US will remain the primary demand driver for spending on analytics solutions, growth will increasingly shift towards Asia in line with trends being seen in IT spending.  India is likely to become a key source of analytics processing power, both digital and human based, China is likely to emerge a large demand center driven by a strong government focus on reforming various industries and aspects of the economy which will require the deployment of analytics solutions.  The talent gap will continue to drive outsourcing and offshoring – both to third party analytics providers as well as to captive centers.  Some of the successful business models which could emerge within this context over the near term are:

  1. The Personalisers: companies which can leverage both technology and strong domain expertise in order to drive consumer insights which help companies drive their toplines and better focus their marketing budgets will continue to drive value for their clients – however increasing automation and a move towards results-based billing could pose challenges to players who cannot innovate rapidly enough.  These players will largely emerge out of the Thinkers and the Counselors and be the fastest growing and most profitable.
  2. The Behemoths: given the large offshoring opportunity, there is a clear opportunity for one or more Indian and Chinese (based) players to emerge as a scaled provider of analytics or another IBM (which itself is one of the largest IT employers in India), which can provide best-in-breed technology platforms and services on a large scale – however it is unclear how long it will be before a consolidator emerges in this industry.  These companies will emerge out of the Arms Dealers and will be the consolidators in the industry.
  3. The Factories:  given that the scope of analytics encompasses everything from research, consulting, servers, networking, software, statistical analysis; there will also be opportunities for offshore analytics providers to ‘productise’ various offerings by leveraging technology and human talent in order to offer them on a much larger scale.  In due course, such players will drive the industry to compete on costs and results.  These players will emerge out of the Do-ers and the Utilities and provide the infrastructure and backbone for the industry.

Given the magnitude of the trend, clearly there may be other successful business models and technologies which emerge.  However, the keys to creating value in the big data ecosystem will be (i) a relentless commercial focus on creating value for clients through analytics, (ii) the ability to attract and train the right mix of human talent, and of course (iii) continuous technological and process innovation.


Greater Pacific Capital is an active participant in the growth of big data analytics through an investment in an Indian provider of outsourced data services and through an investment in a market-leading IT services company in China which is leveraging big data analytics in the Chinese market.

For further information contact Nandan Desai at nandan.desai@greaterpacificcapital.com

Nandan is a senior member of the firm’s Indian office. Previously, he was a Vice President at New Silk Route, an Asia-focused growth capital investment firm where he was part of the team that established the Mumbai office. His sector experience includes telecommunications, IT and knowledge services, financial services, infrastructure, industrials, and consumer goods. Prior to New Silk Route, Nandan was an Analyst at Warburg Pincus in New York where he worked closely with senior members of the firm on macroeconomic strategy. Nandan started his career as a journalist and macroeconomic consultant and has a BA in Economics from the University of Chicago.

[1] Alvin Toffler published The Third Wave in 1980 – which introduced the concept of the post-industrial “information age”

[2] Source: IDC, Worldwide Big Data Technology and Services Forecast, 2012

[3] Source: IDC, includes Big Data services (c.40%), software (25%), storage (20%), and servers / networking (15%)

[4] Amazon has developed an industry-pioneering recommendation engine which has allowed it to grow at 30% over the last decade – from US$5bn of sales in 2003 to US$75bn currently; source: Fortune (http://tech.fortune.cnn.com/2012/07/30/amazon-5/)

[5] IBM puts it well, “big data inspires new ways to transform processes, organisations, entire industries and even society itself.”

[6] Source: IT spending forecast for 2014 from Gartner Research (January 2014) and global (nominal) GDP estimates from IMF World Economic Outlook (October 2013)

[7] Source: Martin Feldstein, Why is Productivity Growing Faster?, Working Paper 9530, National Bureau of Economic Research (NBER), February 2003, available at: http://www.nber.org/papers/w9530.pdf?new_window=1

[8] Source: IDC, Gartner, industry reports, GPC analysis

[9] Source: McKinsey, Big Data: The Next Frontier for Innovation, Competition, and Productivity (June 2011)

[10] See our Sign of the Times article from July-2013 on Cyber Attack, Defence and Security in the Making and Preserving of Superpowers