Finding the Diamonds in Big Data

What is Big Data and why is it suddenly becoming such a ‘hot topic’ for businesses across all sectors? This article aims to demystify this ever increasingly talked about subject and explores the real value to organisations. It looks at the talent profile and how businesses need to overcome the challenges of a very narrow global pool of expertise now in order to reap the longer term benefits.


A lot of people continue to be mystified by Big Data, what it is, and why it's suddenly become such a 'hot topic' for businesses across all sectors globally? When clients ask about hiring in this area, it's clear there is a lack of clarity on the talent profile, the location of skills and how they can identify and attract the best talent.

90% of digital data is unstructure

What is evident is that demand for skills in this area is rapidly on the rise. To understand the origins of this pressure it would be useful to start with some context, a quick and basic explanation of Big Data and why it presents both an urgent challenge and huge opportunity for organisations.

At a very simple level, Big Data can be defined as a vast collection of data that is uncategorised and not easily searchable. In the last few years there has been what is commonly referred to as a 'data explosion': a sudden rise in available data caused by the rapid increase in data sources from media and communication outlets, social media and crucially machines, with the advent of IoT and smart devices. In effect, this has created vast lakes of information, much of which is generated in real-time, meaning it's not just the volume of data, but also the speed at which the data is generated, variety of sources and the unverified accuracy that create the unique characteristics of Big Data.

Add the additional complexity that 90% of digital data is unstructured and it's easy to understand why this makes an unprecedented challenge for organisations. In essence they are now sitting on vast quantities of rich, potentially powerful, yet invisible information.

The role of the data science/ big data function is to create value from Big Data through the application of highly advanced technologies and scientific methodologies.

Telematics insurance ……monitoring real-time behaviour allows the insurer to assess risk and create bespoke pricing and products accordingly.

So what is the value to organisations?

Big Data Analytics is changing our world: in healthcare it's helping us predict disease patterns and find cures, in science it's helping us make new discoveries and in security it's helping the police predict crime patterns and analyse criminal behaviours.

In business it's allowing for faster, better and more qualified decision making: you'll hear talk in the corporates about transformation to a more 'information centric' culture and the more analytics driven businesses in technology and telecommunications going one step further, with the Chief Data Officer (CDO) or Head of Data Analytics being positioned as the top strategic role within the organisation. An interesting use of Big Data analytics is to create new products and services for customers. A good example of this is telematics insurance (also known as black box insurance), whether that be a sensory device in a car, the home or your pet, monitoring real-time behaviour allows the insurer to assess risk and create bespoke pricing and products accordingly.

The opportunity Big Data presents to deepen the customer relationship through providing personalised advice and products to customers is significant. Showing value to customers through an enhanced customer experience is the key to reversing the 'switch to save' behaviour we have seen across all sectors in the last few years.

Enhanced customer experience is the key to reversing the 'switch to save' behaviour.

Cost reduction is also key. Insight driven approaches can transform a huge range of organisations' operational processes to bring immediate benefit. Within utility companies, where, with the advent of smart homes and the IoT, almost every aspect of the network will be visible, companies can get a comprehensive view on when and how customers are using devices across their network, for example, kettles being switched on in the X factor ad break, meaning demand planning models can be much more sophisticated.

Issues can also be detected before the customer has even noticed and organisations can have them pre-emptively resolved, equating to greater efficiency, fewer fines and complaints, thereby giving a more seamless customer experience.

Profiling talent

So what are the skills and experiences required to bring value to the Big Data challenge? Broadly speaking, senior roles in this space can be broken down into three categories; 1. Lead Data Scientists, 2. Big Data Analytics leadership roles, and 3. Chief Data Officers. The latter being a broader leadership role that is relatively new in concept and one we have seen rapid uptake of by organisations in recent years.

The common theme between the roles is the level of intellect and qualifications that is required to truly excel in this area is significant, in fact it's pretty much akin to that of a Rocket Scientist! As a bare minimum organisations who are building serious analytics capabilities will need a Data Scientist with either a PhD in Artificial Intelligence, Computer Science or a Mathematics and Science subject, which involves working with novel methodologies to interpret complex data sets.

The data explosion and upsurge in digital usage is putting huge pressure on talent supply: the existing pool of talent globally is far from adequately meeting demand.

Whilst candidates in this space tend to have 'off the scale' IQs, recruiting individuals with strong EQs as well tends to be the biggest challenge. The fact remains that individuals with extremely strong analytical, mathematical and scientific capabilities can often be lacking in emotional intelligence. In the field of Big Data, having a healthy balance of both is required at leadership level: the IQ to create the value out of the data and the EQ to translate the key messages effectively to the business, thus having the ability to present a compelling story and the actionable insights to make the value meaningful.

Location of skills

The data explosion and upsurge in digital usage is putting huge pressure on talent supply: the existing pool of talent globally is far from adequately meeting demand.

The UK has suffered from a traditionally low concentration of technical and analytical skills. With many organisations favouring outsourcing offshore as a quick fix solution to the skills deficit, we have seen a lack of investment and intent to home grow these skills. Whilst there is a small pool of high quality talent, competition for the best people remains fierce.

The low concentration of skills has had a number of consequences across the talent landscape for senior leadership roles. Firstly, the creation of artificially high salaries; candidates with limited experience and lacking in well-rounded business skills are often over promoted into roles and attracted by the 'highest bidder' into senior positions, essentially putting middle-senior level managers into Director level roles and enjoying executive packages.
Secondly, candidates are headhunted frequently and as such the candidate market has become highly transient, with senior candidates often unable to evidence having delivered sustainable transformation before moving on to another role.

Critical mass of talent can, unsurprisingly, be found in the digital goldmine of the US West Coast or India.

On a global level the critical mass of talent can, unsurprisingly, be found in the digital goldmine of the US West Coast or India, a long established hub of technical outsourcing. However, even with the concentration of good talent in both locations, skill shortages are still projected. A recent McKinsey report in the US forecasts a shortage of 180K Data Scientists and 1.5m analysts and managers by 2018. In India skill supply is actually limiting growth; not only are they continuing to deal with the sharp rise in outsourcing demand but are also battling an internal digital explosion whereby internet and mobile phone use is increasing at an unprecedented rate.

Business solutions

So what can organisations do to attract these skills? In the short to medium term it's clear that any organisation that's serious about attracting top tier talent, needs to take a truly global approach to their search. In addition, being prepared to relocate individuals and 'buyout' elements of their existing package is also key; ultimately organisations who have managed to attract the best talent are now working hard to retain it and when going into battle for such skills, businesses need to be armed with the appropriate weapons to win. As well as hefty packages, significant LTIP buy-outs running from £300K - £800K are not uncommon when we're dealing with premium talent in this space.

In addition to having the right financial incentives, the right armoury in this case also includes the 'intellectual challenge'; a strong articulation of the complexities and nuances of the data landscape and how this relates to the role accountabilities. This is critical in motivating candidates who ultimately are motivated to break through new boundaries and navigate previously uncharted waters in the field.

In terms of long term solutions, 'home-grown' talent through interventions in academia is becoming a popular solution to help grow and secure a strong pipeline of talent. Whilst academia has become a popular sourcing ground for corporates looking to strengthen their data leadership population, foremost technology and telco organisations are now starting to sponsor talented graduates through their postgraduate qualifications in order to attract and secure the best and brightest talent for the future.

Therefore, to get ahead in the future businesses firstly need to acknowledge and own the problem but more importantly must put in place interventions now in order to stem the deficiency of skills later on. This isn't an issue that can be solved purely by buying in talent right now just to address immediate business issues, the problem must be looked at over the longer-term as well. Not just programmes of acquisition but grass roots programmes of development through universities and colleges, after all Big Data won't go away and exploited correctly, will in fact become the life blood of many organisations.

In summary

We live in a world full of data, in fact full of Big Data, but data is only useful once analysed, interpreted and delivered in an easily digestible form. No matter which sector we work in, success seems to follow whomever keeps informed and ahead of changing demands. Businesses need to put themselves in that position, by using Big Data to derive the next trend, create the next 'thing' or deliver the next product or service innovation. Whatever that might be in the future it needs real people to capture, analyse and make sense of the data that's hitting us from all manner of sources at an alarming and ever increasing rate. Without investing in the right talent now businesses will only ever be challengers in their respective markets.