Big Data in Retail
August 2013
Big Data in Retail Overview Today, retailers are busy tackling the effects of the global slowdown, and at the same time face a big challenge in the form of effective and efficient data management. Retail sector generates large volumes of data across the supply chain and across various customer touch points through its omni-channel operations. At the same time, the current digital customer and social media experience add to this massive data explosion, making it hard for retailers to manage huge volume of data. But a fair analysis of these huge volumes of Big Data presents a sizeable opportunity for retailers. Moreover, with the use of advanced technologies to understand the trend behind the data, retailers can maximize their potential and win in the competitive marketplace. This white paper highlights how Big Data is fast becoming the solution retailers can adopt to maximize revenue in the current state of the retail industry. The paper also discusses the key pillars of Big Data along with its transformative opportunities and an adoption roadmap of using Big Data technologies to help retailers make better decisions on their marketing campaigns, merchandising and supply chain management.
Understanding Big Data Definition of Big Data varies by sector, depending on what kinds of software tools are commonly available and what sizes of datasets are common in a particular industry. But in general, “Big Data” refers to large collection of complex data sets that are difficult to process (capture, store, manage and analyze) using on-hand database management tools or traditional data processing applications. Big Data – Industry Definitions “Big Data is high-volume, velocity and variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making.” –
Gartner
“Big Data is a new generation of technologies and architectures designed to extract value economically from very large volumes of a wide variety of data by enabling high velocity capture, discovery and analysis.” –
IDC
“Big Data is data that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or does not fit the structures of existing database architectures. To gain value from these data, there must be an alternative way to process it.” – Source: Gartner, IDC and Others
O’Reilly
In today’s global scenario, Big Data has become a torrent flowing into every areas of business. It poses a major opportunity for CIOs to drive added value for their business, by deriving insights and identifying patterns from the huge amounts of data available. In short, it is neither a technology in itself, nor a distinct and uniquely-measured market of products or vendor revenue. Instead Big Data is termed as a technology phenomenon that comes into picture with the rapid rate of data growth, complex new data types and parallel advancements in technology. According to the 2011 IDC Digital Universe Study estimates, in 2005, 130 exabytes of data were created and stored, which further grew to 1,227 exabytes in 2010. The study estimates that by 2015 the amount of data created and stored is projected to grow at 45.2% to reach 7,910 exabytes. Figure 1 below highlights a decade of digital universe growth in terms of data storage. Figure 1
A Decade of Digital Universe Growth (Storage in Exabytes)
2005
2010
130
1,227
2015
130
Source: Info-communications Development Authority of Singapore
Characteristics of Big Data – The Three Vs Most of the definitions suggest that volume forms a central issue when it comes to Big Data and extreme information management, but that is not true. Along with volume, velocity of data generation, variety of data types and underlying complexity issues form the complete spectrum of Big Data and extreme information management. Together, these three attributes (volume, variety and velocity) form the three Vs of Big Data and constitute a comprehensive definition. In addition, each of the three Vs has its own ramifications for analytics. (See Figure 2)
Figure 2 VOLUME • • • •
• • • •
Terabytes Records Transactions Tables, files
3 Vs of Big Data Batch Near time Real time Streams
VELOCITY
• • • •
Structured Unstructured Semistructured All the above
VARIETY
Source: TDWI Research
Volume: Volume forms an integral part of Big Data, and the ability to manage it in large numbers is a key requirement expected from a retailer. Also, volume is a relative term – most people define it in terabytes and sometimes, in petabytes. It all depends upon the size of the firms – smaller firms are likely to have mere gigabytes or terabytes of data storage as opposed to big global enterprises that have petabytes or exabytes of data. In practice, firms store data of all sorts – ranging from financial data, tables, medical data, environmental data and so on – which will go on in future, hence adding data piles from terabytes to petabytes, exabytes, or more. According to IDC Retail Insights estimates, a total data base of 9.6 petabytes was held in social media in 2012, with another 10.2 petabytes expected to be added in 2013. Also, the estimates conservatively say that 19.8 petabytes of created content drives 14,000 petabytes to 18,000 petabytes of content consumed. Variety: Typically, data comes from a variety of sources (including both internal and external to an organization) and in a variety of formats. Moreover, with new technologies such as sensors, smart devices and social networking platforms, data has become more complex in nature to analyze as it includes not only structured traditional relational data, but also semi-structured and unstructured data. (See Figure 3)
Figure 3
Data Volume
Unstructured Data (e.g. Video, rich media etc)
Semi-Structured (e.g. Weblogs, social media feeds)
Data = Big, Complex, High Velocity & Wide Variety
Structured (e.g. sensor, operational data, data warehousing information)
Time
Source: Info-communications Development Authority of Singapore
Structured data: Data formats that are grouped into a relational scheme (e.g. rows and columns within a standard database) are referred to as structured data. The data configuration and consistency allow it to respond to simple queries to arrive at usable information, based on a firm’s parameters and operational needs Semi-structured data: Data formats including weblogs and social media feeds that do not conform to an explicit and fixed schema. This data is categorized as self-describing and contains tags or other markers to enforce hierarchies of records and fields within the data Unstructured data: Data formats such as images, audio and video files, which cannot be easily indexed into relational tables for analysis or querying Velocity: One of the other characteristics by which Big Data is defined is velocity. The conventional understanding of velocity typically considers how to make real-time decisions based on a large number of data points available in an instant. It also considers the pace at which the data is gathered and retrieved. In general, velocity is applied to data in motion – the speed at which the data is flowing.
Big Data in Retail With global economic slowdown and high costs in place, global retailers are already battling hard to increase their share of the customer pockets. Moreover, they are also striving hard to improve efficiencies and reduce costs across their value chain by operating within tight singledigit percentage margins. In addition, the advent of a new generation of information economy driven by digital customers demanding convenience, personalization, and promotion-based pricing has made thriving in the current setup even more challenging for retailers. As the digital world continues to emerge, customers have started purchasing products across multiple channels – such as mobile and web. Moreover, they use real-time information through social media, customer forums and blogs, and make purchase decisions through ratings, reviews, price comparisons and product recommendations. In such transactional process, customers generally leave a trail of information about their preferences and behavior. This information creates a huge chunk of data that retailers need to assess and follow to find their way in the competitive market. However, analyzing such huge and diverse sets of data to better understand consumer behavior has become a huge challenge for retailers today, which will continue to grow in the future. A study by Gartner shows that data volumes are estimated to grow 800% by 2017, of which 80% will reside as unstructured data. Such an increase in data presents both an opportunity and an asset for retailers, only if they can make sense out of it. That is the reason why retailers need solutions that can help them access their customer and product information in order to earn customer loyalty, launch successful new products in market, collaborate through supply chains with business partners, enable associates, reduce risk, ensure compliance, and above all, burnish their brand in a highly competitive market. Such solution needs to quickly gather and process huge chunks of data from multiple sources like social media, customer data, market data and supplier data; and draw real-time insights and analytics to enable quick decision. Today, retailers have started to move slowly and cautiously to harness the power of Big Data. According to a study conducted by Edgell Knowledge Network (EKN) in North America, close to 80% of the retailers were aware of Big Data, but the depth of understanding was still low. Also, about 47% of the retailers do not even have an understanding of Big Data's implication on their business. The study also highlighted that about 57% of retailers already have a Big Data strategy in place or are building one. However, only 22% and 30% of retailers have executed or are executing a Big Data proof of concept
"Big Data is a real opportunity, and smart retailers are cautiously setting themselves up for success. If retailers can avoid getting caught up in the semantics of Big Data definitions, and focus instead on what decisions are valuable to their business, Big Data can be the game-changer it is touted to be." >> Gaurav Pant, Research Director, EKN
(POC) and a Big Data project, respectively. This indicates that retailers are still in a nascent stage to build an understanding of the implications of Big Data.
How Big Data Empowers Retailers The most important factor for retailers to succeed in this challenging setup is to build an understanding of the consumer habits. Retailers are digging consumer data and using Big Data technologies to analyze such information and then strategize their different activities such as marketing, customer service management, merchandising & supply chain management and so on. Some of the major benefits that retailers draw using Big Data are listed below. See figure 4 Merchandising & Supply Chain Management Real-time Delivery Management – Although the goods shipment process has improved over the years, Big Data helps it improve further by providing a real-time feed on weather, traffic and truck location to determine the exact time of delivery. Such features benefit retailers who are shipping perishable or expensive products or who need to keep track of their shipments for some other reason like customer delivery appointments. Moreover, such solutions can be implemented without making significant changes to the existing supply chain model as most of the work is focused on integrating multiple data feeds. Effective Order Picking Process – Big retailers are using automated mechanism to pick and ship orders faster for better order fulfillment. When it comes to small retailers, order picking becomes a labor-intensive process. But with the implementation of Big Data, smaller retailers too can improve their order fulfillment. Big Data solution allows retailer to analyze data from multiple sources like orders, product inventory, warehouse layout, and historical picking times that helps improve the entire order picking process. The solution also helps improve the inventory management both in store and warehouse for smoother operations. Vendor Management – Retailers generally work with multiple vendors in their supply chain. Big Data analytics help in real-time management of these vendors by reviewing their performance against a set of key performance indicators (KPI), including vendor profitability, on-time performance, and customer feedback and complaints. The KPIs are tracked in real-time by integrating with vendor systems, financial inputs like cost of goods, social network feeds on vendor deliveries and product packaging. These real-time analytics solutions ensure that the quality of service and profitability of the vendor business meet desired standards without putting in any extra effort. Anticipate Demand – One of the key challenges that retailers often face is to gauge demand accurately. This can even lead to loss of revenues. For example, in 2012, Marks & Spencer (M&S) launched a new meek knitwear line during peak season, for which it underestimated the potential demand. This resulted in stock-out situation at M&S, which contributed to their
abysmal sales. Big Data comes handy in such situations; it provides retailers a real-time view of the product demand, product sales and sourcing process. Moreover, Big Data also help retailers stop marking certain products as "backordered" as they are well aware of the exact lead times for sourcing such products. This also results in a reduced number of abandoned shopping carts and queries about the order. Marketing Consumer Segmentation – In a traditional retail setup, retailers usually segment consumers based on some basic macro variable factors such as age, gender, life stage, store visits and basket size. But in today’s scenarios, they need to know more about consumers. Such information not only help retailers better understand the consumer need but also help them in framing an effective and efficient marketing and promotional strategy to avoid mass promotions and discounting. POS-based Promotion – Traditional promotional offers are conducted through direct mail, email and vouchers, which have very low conversion rates, and many times retailers are not able to take an appropriate decision on the promotional activities. Big Data analytics provide retailers very precise information about consumer to offer promotions at point of sale (POS) through instore analytics and other self/mobile check-out devices that help retailers improve the conversions. Designing Marketing Campaigns – In the current scenario, a retailer’s campaign strategies are not liquidating a consumer’s information into a successful marketing campaign and sales generation. Retailers need flexibility to strategize their campaign to target the right audiences through deep analysis of market trends and customer behavior. Big Data solutions help retailers in providing a 360 degree customer view and analyzing customer actions, which, in turn, helps in designing an effective campaign to maximize the results. Customer Service Management Engaging Customers for New Product Development – With the increasing demand of customers for personalized products and services, retailers have started using different techniques – like co-creation – to attract and engage the customer with their brands. For example, Nike (NikeiD) has developed an online app “build your own shoe” for customers to create their own shoe design on its website. Such customization and social media initiative resulted as the strategic revenue generators for Nike. For fiscal 2010, Nike’s web sales increased 25% to about USD260 Mn, up from USD208 Mn in 2009. With Big Data, retailers can analyze customer interactions across all channels to understand their need and also add a new product line of consumers’ choice. Product Recommendation and Customer Satisfaction – Big Data helps retailers make an instant product recommendation and advertisement to the consumers while shopping. Amazon is one
such retailer, which generates more than 25% of sales through its recommendation engine that suggests popular products to the consumers instantly. Recommendations help retailers improve their merchandise sales and associated services, hence creating a strong customer loyalty. Big Data also helps retailers leverage out the unused data such as surveillance videos, to improve the shop floor designs and customer satisfaction. Figure 4
Source: Secondary Research
Others Diversifying Into New Business Verticals – Often retailers are indirectly linked with other industry verticals including banking, financial, media services, digital content and etc. Big Data enables to combine the entire customer data together across all the verticals. This not only helps enterprises better cross-sell and up-sell capabilities but also helps them create a better value proposition for customers.
Big Data Adoption Strategy Every retailer wants to improve margins and drive supply chain efficiencies. That is the reason why retailers carefully avoid solutions that require huge overhaul, as most of them want to avoid high cost investment. Today, retailers want to implement cost-effective data management solutions with a proven successful adoption methodology. A successful adoption strategy for Big Data in retail involves the following steps;
Build Strategy Retailers are aware about the benefits of Big Data and wish to incorporate this opportunity in their day-to-day operations. They are willing to test the impact of Big Data solutions by implementing it in their existing IT setup, so that its benefits and complexities can be gauged. As a first step to start, a clear cut strategy needs to be framed to adopt Big Data, which typically starts with a pilot execution. Execute Pilot To execute pilot, retailers first need to identify a small business unit or a department on which a pilot can be conducted. Moreover, the executive management team should also assign measurable KPIs and solution objectives of the business to set standard benchmark and give a clear proposition. Once the above has been assigned, then run the pilot program. Large-scale Implementation After the successful test run of pilot program in one business unit, retailers can implement the solution adoption process across all the relevant businesses, including customer analytics, product development, inventory management, etc. Also, retailers need to adopt a top-down approach to spread awareness about the successful implementation of Big Data analytics in the system. Solution Management Once the solution has been implemented, it becomes a mandate to introduce measures to ensure that solution implementation is being managed in an efficient and proper manner. Retailers need to deploy additional solutions to control the flow of information in a controlled and precise manner to enable them to communicate across all the business lines. Data-driven Enterprise Adaptation of Big Data adds to the transformational value of the enterprise, which helps liquidate the information into top-line of the business, hence leading to a data-driven enterprise. Initially, retailers adopted Big Data solution to reduce the operational expenses, but it is advisable to take a more holistic approach that adds value to the top-line. Source: Infosys
Case Study – Big Data Helps Tesco Save EUR20 Mn Annually Objective In an attempt to save over EUR20 Mn annually, Tesco (UK’s largest retailer) came up with a plan to cut its refrigeration energy costs by up to 20% across 3000 stores in the UK and Ireland. To achieve the savings, Tesco implemented sophisticated business intelligence technology to ensure its refrigerators operate at the right temperature. Initial Observation After analyzing gigabytes of refrigeration data, the sophisticated computer systems highlighted that various Tesco stores in Ireland were running their refrigerators at a lower temperature (-21°C and -23°C) than necessary. Realizing the situation in 2010, Tesco started to work closely with refrigerator manufacturers in collecting refrigeration data from in-store controllers and delivering the same to a dedicated data warehouse via internet. The data warehouse took readings every three seconds from in-store sensors, processed the data in real-time, and displayed the results on a Google map showing performance of refrigerators at more than 120 Irish locations. It also monitored and controlled the performance of Tesco's heating and lighting systems. Pilot Execution In 2012, Tesco teamed up with IBM to look at the data in more depth, which also helped the retailer identify the potential savings in its refrigeration costs. IBM researchers analyzed refrigerator data – equivalent to 70 million data points – collected from one of the Irish stores over the course of a year. They also developed a set of KPIs that looked at data aggregated every month over a year. They used standard blade servers running IBM's SPSS statistical software package to analyze the collected data. Once implemented, the six-month pilot project helped Tesco make immediate savings in the trial store. It helped the retailer in identifying refrigerators that were operating at a lower temperature than necessary or behaving in an anomalous way. Large-scale Adoption After the successful pilot execution, Tesco plans a six-month trial with a larger number of stores in the second phase. Once the second phase is successful, Tesco will further roll out the energy-saving project to more than 3,000 Tesco stores across the UK and Ireland within 18 months. The retailer will potentially save EUR20 Mn, if the full-scale roll-out exhibits similar results as it did during the pilot. The project also helps the retailer in reducing its maintenance costs. Displayed results of the processed data on the Google map alert engineers in identifying refrigerators that are not working at the optimum temperature. This helps the maintenance department check the refrigerators remotely, diagnose the problem, and turn up with the right part. Earlier, engineers who came to the store to diagnose the problem had to return to the depot to collect the equipment needed to fix the problem.