A FUTURE WITH DATA
THE BLACK SWAN GUIDE
2
HELLO! Welcome to a few of our thoughts on retail
We’d be lying if we told you that four years ago when we started Black Swan we’d be using the power of the internet to help the marketing and supply chain work better. We’d noticed that every opinion, invention and decision you’d ever need to make existed on the internet somewhere – but we didn’t have a clue where that would be useful.
So, A Future With Data is designed to tell you a little bit about what we can do, and contains a bit about some of our successes so far. Our techniques are already saving our customers millions; and we are also helping to give a little back by sharing some of these techniques with not-for-profits for free. Although this is a relatively new field, positive, measurable results are making it one of the fastest growth areas in data analytics.
Steve King & Hugo Amos Co-founders
3
THE DATA ECONOMY COMES OF AGE It turns out prediction is simpler than we thought. If we stop trying to look for reasons, the future is staring us in the face… by Rose Barnett (Data Science Consultant) The ubiquity of Big Data is such that
upon which the business took action,
Gartner dropped it from their “Hype
with quantifiable commercial results
Cycle of Emergent Technologies” back
(preferably expressed in one of the
in 2015. Across sectors, businesses
major world currencies). You might
are scrambling to make every function
just detect a conspicuous absence of
‘data driven’, and there’s no shortage
concrete case studies to validate this
of firms lining up to help them. The
‘data-insight-action-value’ chain as a
Data Analytics industry, dedicated to
concept.
helping big businesses leverage the petabytes of information they now generate and store, is worth $122bn and growing.
popular among jaded office workers in the mid-2000s, players would seek out examples of bloggers who made
The basic premise of the industry’s
so much cash from blogging that they
offering is this: hidden in that huge
quit their jobs to blog full time (at
mass of enterprise data are latent
home, in a hammock, with a Daiquiri).
patterns. If you could only interpret
Veterans players eventually noticed
your data properly, like an explorer
that there is only one blogging topic
deciphering an ancient scroll, you’d
lucrative enough to support such a
be able to unearth these precious
lifestyle change - How To Make A
business secrets. Specialist analytic
Living From Your Blog So You Can
software tools are needed to crack
Quit The 9-5.
the code. Into these go big, diverse, disparate, messy data and ‘actionable insights’ come out.
Clicking through pages ‘unlock the value of your big data!’ advertorials, a cynic might suspect that the best
Here is a game you can play yourself
(and perhaps only) method of deriving
at home: search online for a real-
value from big data is to go into the
world story of how big data analytics
business of telling people how to get
produced a piece of ‘hidden’ or
value from their big data.
‘unexpected’ intelligence, based
4
In the original version of that game,
A cynic might suspect that the best method of deriving value from big data is telling people how to get value from their big data All that’s happened is that technological
Firstly, in order for a puny human brain
innovations in data handling capability
to interpret large and complex datasets,
(made by companies like Google to
they must first be made ‘smaller’ via
deal with the scale and complexity of
aggregation, summarization, description
Web 2.0) temporarily leapt ahead of
and presentation, which kind of misses
our progress in learning how to apply
the point.
them – progress we make through experimentation.
Secondly, there’s just a natural limit on how far having information about your
In the interim, firms have defaulted to
business is going to help you make it
leveraging big data in exactly the same
work better. Any enterprises’ data is
way they previously used small data:
simply the digital impression left behind
for reporting and business intelligence.
by real-world transactions. Typically,
Having invested in purpose-built
mining that internal data will validate
tools to analyse data at scale, they’ve
basic hypotheses upon which the
been rewarded with cool interactive
business is predicated (“we make profits
dashboards visualizing it. These are
in our luxury fashion stores when they’re
basically auto-generated charts,
located in affluent areas”). In the worst
conspicuously similar to the manually-
case, it can make you uncomfortable
created Excel and PowerPoint reports
by totally undermining those core
executives were staring at back in 2005,
assumptions without suggesting a back-
but far prettier and costlier. It’s easy
up plan – (“we thought people bought
to see why this approach hasn’t quite
ice-cream on impulse when it’s hot
delivered on the big data promise.
and sunny outside; turns out we were
There’s more to data than visualisation.
wrong”).
5
THE DATA ECONOMY COMES OF AGE Knowing how to rig the game so that the computer easily wins is the most important trade secret in applied prediction Big businesses have absorbed
The beauty of predictive algorithms
Google-style tech, but are only
is that they don’t need to understand
just beginning adopt Google-style
the cause-and-effect behind statistical
thinking alongside it. Machine-learned
relationships in order to work very,
translation algorithms, made possible
very well in practice. For an enterprise
by the availability of a massive corpus
to glean the benefits of prediction,
of textual training data and souped-up
it must first give up trying to deduce
processing power, have for instance
why things are a certain way, and start
no conception of French or Arabic
trusting the lines of code which tell us
grammar. Amazon’s recommendation
that they are.
algorithms generate 35% of sales without knowing why certain products are ‘frequently bought together’. It’s
Applied prediction
this very characteristic which makes
Predictive analytics is used to detect
them so powerful – if a machine
fraud and stop cyber attacks, but
can’t judge, it can’t make the errors
it’s largely an unexplored frontier for
of judgement to which humans are
most consumer-facing businesses.
prone.
Misconceptions about what
Algorithms now detect when drilling
‘prediction’ means in this context
equipment in oil fields is about to fail based on thousands of sensor datapoints, enabling ‘predictive maintenance’. Imagine if, instead of applying machine learning to the
the future is just a special case of the general capability. But there’s also an unspoken feeling that using computer models to make decisions is somehow
problem, analysts had compiled
a risky business.
these complex datasets into summary
We can evaluate how accurate
reports, and tried to divine ‘insights’
predictive models are before
about why the equipment breaks
unleashing them to make real-world
so they could attempt to stop it
decisions. We can even choose the
happening.
6
are partly responsible – forecasting
‘type of accurate’ we care about, and
the right ‘indicator’ data was a success
automatically build the best possible
factor - Twitter networks mimic real-
model for that criterion. For most
life social networks, so the spread of a
business use cases, a model doesn’t
contagious bug around a community
have to be terribly accurate before
of people is mirrored there. But what
it’s already beating the competition
really made the difference was the
(namely, the way that decision was
choice of what to predict. Google’s
made before). We can also simulate
algorithm tried to estimate the number
how the old and new methods perform
of people affected by a flu outbreak –
against each other from the safety of a
ours just had to predict the time and
virtual lab.
place. Knowing how to rig the game
On the flipside, it’s entirely possible to make life difficult for yourself when designing a data algorithm. Google’s
so that the computer easily wins is the most important trade secret in applied prediction.
Flu Trends project is often cited as an
We’ve barely scratched the surface
example of ‘when machine-learning
of what’s possible with commercial
goes awry’ - even as a failure of Big
applications of artificial intelligence.
Data itself. The algorithm aimed to
To make progress, business leaders
estimate the prevalence of real-world flu
need to take a step into the future by
cases based on Google search queries,
nominating the parts of their enterprise
trained on historical data about both.
they’re prepared to make truly ‘data
Initially it performed well, but was soon
driven’, and surrendering them to the
wildly over-estimating the number of
power of science.
cases. Machine-learned algorithms are supposed to get better over time, not worse. Black Swan approached the same subject in the context of consumer healthcare. We built a model to pinpoint outbreaks of the sniffles based on geo-located Tweets where users mentioned the symptoms. Choosing
7
“
BY 2018, OVER HALF OF LARGE ORGANIZATIONS GLOBALLY WILL COMPETE USING ADVANCED ANALYTICS AND PROPRIETARY ALGORITHMS, CAUSING THE DISRUPTION OF ENTIRE INDUSTRIES.
(GARTNER 2016)
”
WHAT IS PREDICTIVE ANALYTICS? It’s something that’s been with us almost since time began. So why is everyone talking about it now? by Dick Fear (Director of Data Science)
Like an avalanche where no one flake of snow can be pinpointed as the cause, the connected world has reached a sort of critical mass
You can think of predictive analytics in
forecast sales and understand
much the same way as physics. Just as
consumer behavior since the dawn of
physics can be viewed as the process
commerce began.
of applying mathematics to the natural world, so predictive analytics is the business of applying maths to, well, business. When we apply maths to a system, it can give us a better understanding of why and how things happen, and that in turn lets us predict those things happening again in the future. It’s been around in one form or another for hundreds of years, as evidenced by the rose diagrams, or ‘coxcombs’, of Florence Nightingale [See right], or John Graunt’s efforts to model the spread of bubonic plague in 1662 [See overleaf]. Similarly, businesses have been attempting to
10
So why is it that the concept of predictive analytics suddenly seems revolutionary? It is not thanks to any one particular invention; but like an avalanche where no one flake of snow can be pinpointed as the cause, the connected world has reached a sort of critical mass. We’ve been hit by a kind of perfect storm of open data movement with the ‘API (Application Programme Interface) economy’, worldwide internet connectivity, a huge growth in computing power, and new analysis techniques. APIs let us bring knowledge systems together.
The lady of the data Florence Nightingale is widely known for her compassionate and effective approach to nursing, which brought the profession into a new era based on the difference that can be made by simple sanitation. However what is perhaps less widely recognized is that the enormous impact she made was born out of the world of statistics, gathering data and gleaning insight. She kept meticulous records of the death tolls in hospitals, and the graphs that she created highlighted the reasons for mortality, showing the dramatic difference the Sanitary Commission made when they were sent out in the middle of the Crimean War.
11
WHAT IS PREDICTIVE ANALYTICS? We have the tools and data, but if a pattern doesn’t exist, we can’t find it
At its heart predictive analytics is a
year. Rather, we found that a whole
simple yet powerful concept. Because
set of other conditions first had to be
of this, case studies can hugely varied,
met, such as various social factors and
from forecasting sales to modelling
that the sunny weekend had to be
social trends, and anything in between.
preceded by a number of particularly
Where data exists, predictive analytics
bad weekends.
can be applied. A great example of this is the major UK retailer who came to us with a recurring yearly problem: Every summer, there is a particular weekend when Brits, as a collective whole, decide that summer has finally started. Naturally this means rushing to the shops on a Friday to buy unreasonable quantities of BBQ food. If a retailer is not prepared this can mean millions in either lost sales or wasted stock. Surprisingly, this weekend does not always fall on the first hot and sunny weekend of the
12
None of this is to say that predictive analytics is a magic wand, more a case of pattern finding in data, and we now have the tools and data to find them where they exist. But if it doesn’t exist – we can’t find it. ‘Data mining’ is a good metaphor for this – so if there is no gold in the ground, even the best excavator in the world won’t be any more use than a shovel.
A picture of the plague Around the beginnings of modern London in the 17th Century, John Graunt, one of the earliest fellows of the Royal Society, brought a new rigour to the recording of scientific data. Until then records of the black death had been wildly inaccurate, not least because the ‘searchers’ appointed to record the deaths tended to be unreliable characters. They had been given this paid occupation chiefly since it removed a burden from the state to give them other payment. They had little skill at determining the cause of death, and were often paid extra by families so they would not register it as plague, but consumption. John Graunt saw this bias and brought a new methodology, laying the framework for modern demography.
13
CASE STUDY
1
Which products do the public really want and when do they want them, so stock levels and shelf space can be optimised?
THE CHALLENGE Our pharmaceuticals industry client knew that sales of their vitamin and anti allergy brands spike during flu season or when the pollen count is rising. However, traditional media planning can only take a blunt approach to targeting this demand. They wanted to deliver increased relevancy and greater media efficiency through targeted digital media.
THE SOLUTION Black Swan combined the company’s own sales data with external data sets such as search behaviour, weather and social media. This created a powerful predictive model that is able to accurately predict outbreaks of cold and flu even at postcode level. This model was used to develop a programmatic media solution that optimized the content of the media as well as only serving it only at exactly the right time and place.
THE RESULT An impressive 12% uplift in sales YOY, 500,000+ consumers engaged with the flu tracker social application, with a 200% uplift in click through rate, while the geo-targeted media reached 9 million consumers.
15
CASE STUDY
2
Using ingredients in your products that the public dislike can cost you serious money. Here’s how to avoid it
THE CHALLENGE Our global FMCG client wanted to create a tool that would listen in to what the public was saying and robustly quantify shifts and sentiment, relating to ingredients that may already be in their products. They also wanted something capable of informing the decisions they make during R&D, helping them see which new ingredients they should be considering putting in their products.
THE SOLUTION We developed an ‘early warning system’ to track trends across a broad range of public data sets, including Twitter, blogs and forums, news, academic articles and more. We were able to model the likely tipping points of these trends and quantifiably link shifting sentiment to sales data.
THE RESULT A real-time business intelligence solution that has helped inform multimillion dollar product reformulation decisions.
17
THE FUTURE OF DIGITAL MARKETING Six of the key topics and tools smart marketers need to know about right now by Steve King (Co-founder and CEO) Trend-based advertising The automated analysis of trends – based on data from social channels mixed with purchase patterns – can now be realised on programmatical media channels. For instance, the surge of lipstick sales in certain stores on certain days and times, to certain types of consumers, can be tied to a surge of interest in social media locally, and tied to potential purchase moments. Programmatic
a consumer is watching a movie or walking through the store, allowing stores to position goods that maximise profit straight into a consumer’s gaze and make the battle for the shop window into a more scientific war of attention. These tools also allow retailers to reorganise their online experience, to maximise purchase funnels, by influencing the right click flows more efficiently than traditional eye tracking.
advertising allows these moments to be automatically targeted by laser focusing on the type and location of a consumer, along exact time and date of their interest.
Dynamic product placement As video content becomes more and more dynamic and able to be tailored to particular consumers on digital, the surge of technology that dynamically
Algorithmic eye tracking Artificial Intelligence techniques can be used to predict where a consumer’s attention will be focused, both instore and online. Intercepting the consumer’s gaze allows marketers to influence to consumer decisions by getting there first. A new generation of tools, such as Black Swan’s Dragonfly, is now taking over from traditional eye tracking studies however, allowing for real-time attention prediction. This can include the moments when
18
places a product or picks the version of your content based on your profile increases. Tools that allow video streams to be intercepted and given different product fronting already exist, and advertisers are now trying to link those with one-click purchases into online retail sites. As content becomes more portable and is easily embedded and licensed for other nonvideo websites, the online retail store will become a targeted entertainment portal.
Mobile loyalty As mobile payment continues to grow and grow, handsets become loyalty cards. To understand a consumer’s
alone. Eventually, automation will make bidding a lot like the Google or Bing search keywords marketers use right now.
location patterns along with social networks is incredibly powerful when put alongside their purchase habits, and replaces old retail loyalty programmes that only know what people are buying. Apple has had a loyalty wallet for years, but recent software releases are moving towards Application Programming Interfaces (APIs) that allow retailers to tie together the consumer’s purchase, loyalty, offer and behaviour into one place – this will allow for greater consumer understanding and the next generation of discount and loyalty cards.
Internet of Things From the slightly gimmicky ordering of beer from your fridge magnet to Amazon’s Alexa – the Internet of Things (IOT) is already hijacking retail APIs so you can order goods by voice. For a long, long time the innovation teams of fridge, microwave and TV manufacturers have been preaching the idea of your shopping list being decided by your own kitchen devices. Now companies like Amazon and Tesco are beginning to push the art of the possible. APIs are available to integrate with an ever-increasing number of connected
Retail online search algorithms Just as getting their products on Google search is now a day to day task for any marketer, making sure products appear high on a retailer’s search is becoming big business. The algorithms were firstly powered simply by relevance to what consumers had been searching for, but now brands are bidding to get into the retailers’ search algorithms, in the same way that they are now pushing to get into key shelf space. Several media companies already provide an agency service around this facility
devices, and IOT protocols now finally becoming standard – so it will finally be possible for your fridge to automatically order your next pint of milk before you run out. Marketers are yet to begin to try influencing consumer purchasing patterns en masse in these w devices, but the challenge will be identical to the old days of mobile phones, with a vast array of different shapes, standards and pricing. In the early mobile days, the quickest marketers to platform made huge returns on investment, before standardisation opened the marketplace.
19
WANT TO KNOW MORE? IF WE’VE WHETTED YOUR APPETITE THEN PLEASE DO DROP US A LINE AT HELLO@BLACKSWAN.COM OR VISIT OUR WEBSITE: WWW.BLACKSWAN.COM