TDWI_CDI_report_web by Tony Ciotti

four t h qua r t er 2008

T DW I bes t pr ac tices Repor t

C u s tome r Data in t eg r at ion Managing Customer Information as an Organizational Asset By Philip Russom

w w w.tdwi.org

Research Sponsors Acxiom DataFlux HP IBM Informatica Syncsort Teradata Trillium Software

fourt h qua rt er 2008

T DW I bes t pr ac tices Rep or t

C u s tome r Data in t eg r at ion By Philip Russom

Managing Customer Information as an Organizational Asset Table of Contents Research Methodology and Demographics . . . . . . . . . . . . . . 3 Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . 4 Introduction to Customer Data Integration . . . . . . . . . . . . . . 5 Definitions of CDI and Related Concepts . . . . . . . . . . . . . 5 Analytic CDI and Operational CDI Practices . . . . . . . . . . . . 7 Why Care about CDI, and Why Now? . . . . . . . . . . . . . . . 8 Business Issues Concerning the Use of Customer Data . . . . . . 9 The State of CDI . . . . . . . . . . . . . . . . . . . . . . . . . . 11 The Perceptions and Realities of Sharing Customer Data . . . . 11 The Scope of CDI Solutions . . . . . . . . . . . . . . . . . . . 12 The Number of CDI Solutions per Organization . . . . . . . . . 12 CDI Funding, Use, and Maintenance . . . . . . . . . . . . . . 13 Benefits of CDI . . . . . . . . . . . . . . . . . . . . . . . . . 14 Barriers to CDI . . . . . . . . . . . . . . . . . . . . . . . . . 15 Key Features of a CDI Solution . . . . . . . . . . . . . . . . . . . 16 High-Priority CDI Features . . . . . . . . . . . . . . . . . . . 18 Medium-Priority CDI Features . . . . . . . . . . . . . . . . . . 18 Low-Priority CDI Features . . . . . . . . . . . . . . . . . . . . 19 Buy versus Build . . . . . . . . . . . . . . . . . . . . . . . . 21 Applications Integrated with CDI Solutions . . . . . . . . . . . 21 Vendor Tools and Platforms that Enable CDI . . . . . . . . . . . . 23 Database Management Systems . . . . . . . . . . . . . . . . 24 Data Integration or ETL . . . . . . . . . . . . . . . . . . . . . 24 BI Reporting and Analysis Tools . . . . . . . . . . . . . . . . . 25 Data Warehouses and Marts . . . . . . . . . . . . . . . . . . 25 Data Quality Tools . . . . . . . . . . . . . . . . . . . . . . . 25 Third-Party Customer Data . . . . . . . . . . . . . . . . . . . 26 Master Data Management . . . . . . . . . . . . . . . . . . . 26 Metadata Management . . . . . . . . . . . . . . . . . . . . . 26 Vendor Tools for CDI or MDM . . . . . . . . . . . . . . . . . . 26 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . 28

w w w.tdwi.org

customer data integration

About the Author PHILIP RUSSOM is the senior manager of TDWI Research at The Data Warehousing Institute (TDWI), where he oversees many of TDWI’s research-oriented publications, services, and events. Before joining TDWI in 2005, Russom was an industry analyst covering BI at Forrester Research, Giga Information Group, and Hurwitz Group. He also ran his own business as an independent industry analyst and BI consultant and was contributing editor with Intelligent Enterprise and DM Review magazines. Before that, Russom worked in technical and marketing positions for various database vendors. You can reach him at prussom@tdwi.org.

About TDWI The Data Warehousing Institute, a division of 1105 Media, Inc., is the premier provider of in-depth, high-quality education and training in the business intelligence and data warehousing industry. TDWI is dedicated to educating business and information technology professionals about the strategies, techniques, and tools required to successfully design, build, and maintain data warehouses. It also fosters the advancement of data warehousing research and contributes to knowledge transfer and the professional development of its Members. TDWI sponsors and promotes a worldwide Membership program, quarterly educational conferences, regional educational seminars, onsite courses, solution provider partnerships, awards programs for best practices and leadership, resourceful publications, an in-depth research program, and a comprehensive Web site (www.tdwi.org).

About the TDWI Best Practices Reports Series This series is designed to educate technical and business professionals about new business intelligence technologies, concepts, or approaches that address a significant problem or issue. Research for the reports is conducted via interviews with industry experts and leading-edge user companies, and is supplemented by surveys of business intelligence professionals. To support the program, TDWI seeks vendors that collectively wish to evangelize a new approach to solving business intelligence problems or an emerging technology discipline. By banding together, sponsors can validate a new market niche and educate organizations about alternative solutions to critical business intelligence issues. Please contact TDWI Research Director Wayne Eckerson (weckerson@tdwi.org) to suggest a topic that meets these requirements.

Acknowledgments TDWI would like to thank many people who contributed to this report. First, we appreciate the many users who responded to our survey, especially those who responded to our requests for phone interviews. Second, our report sponsors who diligently reviewed outlines, survey questions, and report drafts. Finally, we would like to recognize TDWI’s production team: Jennifer Agee, Bill Grimmer, Denelle Hanlon, Deirdre Hoffman, and Susan Stoddard.

Sponsors Acxiom, DataFlux, HP, IBM, Informatica, Syncsort, Teradata, and Trillium Software sponsored the research for this report.

TDWI rese a rch

©2008 by TDWI (The Data Warehousing Institute ), a division of 1105 Media, Inc. All rights reserved. Reproductions in whole or part prohibited except by written permission. E-mail requests or feedback to info@tdwi.org. Product and company names mentioned herein may be trademarks and/or registered trademarks of their respective companies. TM

Research Methodology

Research Methodology and Demographics Report Scope. This report is designed for business and technical executives who are responsible for planning and implementing a program for customer data integration (CDI). This report helps organizations worldwide understand the current state of CDI, as both a best practice and a technology. It identifies and evaluates common starting points and strategies, with an emphasis on sharing customer data across business units as an enterprise asset. Research Methodology. Most of the market statistics presented in this report are based on the report’s survey. In May 2008, TDWI sent an invitation via e-mail to the data management professionals in its database, asking them to complete an Internet-based survey. The invitation was also distributed via Web sites, newsletters, and conferences from TDWI and other firms. Almost 400 people responded to the survey, though not all respondents answered every question. From these, we excluded the respondents who identified themselves as academics or vendor employees, leaving the surveys of 365 respondents as the primary data sample for this report. TDWI Research also conducted numerous telephone interviews with technical users, business sponsors, and recognized experts in the field of CDI. TDWI received product briefings from vendors that offer products and services related to the best practices under discussion. Survey Demographics. The wide majority of survey respondents are corporate IT professionals (62%), whereas the remainder consists of consultants (23%) or business sponsors/users (15%). We asked consultants to fill out the survey with a recent client in mind. The financial services (18%) and consulting industries (13%) dominate the respondent population, followed by insurance (8%), software (8%), manufacturing (7%), government (6%), retail (6%), telecommunications (6%), healthcare (6%), and miscellaneous other industries. Most respondents reside in the United States (68%), Canada (8%), or Europe (8%). Respondents are fairly evenly distributed across all sizes of companies and other organizations.

Position Corporate IT professional

62%

Consultants

23%

Business sponsors/users

15%

Industry Financial services Consulting/professional services

18% 13%

Insurance

Software/Internet

Manufacturing (non-computers)

Government (all levels)

Retail/wholesale/distribution

Telecommunications

Healthcare

Education

Utilities

Pharmaceuticals

Media

Transportation/logistics

Aerospace

Other

Geography United States

68%

Canada

Europe

Australia

Asia

Central/South America

Africa

Middle East

Other

Company Size by Revenue Less than $100 million $100–500 million $500 million–$1 billion $1–5 billion $5–10 billion More than $10 billion Don’t know

17% 19% 11% 16% 11% 15% 11%

Based on 365 respondents. Percentages total 100%.

w w w.tdwi.org

customer data integration

Executive Summary

Who are your customers? Which products and services are they buying across your enterprise? How much business have they transacted with your enterprise so far this year? Where do they conduct business besides your firm? Customer data integration (CDI) helps you answer key business questions.

If you don’t have an enterprisewide solution for customer data integration (CDI), it’s unlikely you can answer any of these questions with a respectable level of accuracy. Therefore, you don’t know who your most profitable customers are, and, in turn, you don’t have a viable strategy for retaining and growing them. If a customer is in peril, you won’t know it until they churn—and maybe not even then. If you’re not acquiring customer data from third parties, you don’t have a complete view of your customers’ financial brackets, consumer demographics, and lifestyle behaviors—information that is essential to successful up-sell and cross-sell, plus accurate customer base segmentations.

Customer data’s greatest value is achieved when it becomes an organizational asset.

CDI helps with these issues by collecting customer data from disparate enterprise and third-party sources and integrating the data into a 360-degree view of each customer that’s complete, up-to-date, cleansed, and standardized. TDWI’s position is that customer data is an organizational asset that’s best shared and leveraged across every business within an enterprise. For customer data to achieve status as an enterprise asset, its ownership and management must be centralized, although physically relocating the data to a central location isn’t necessary. Once centralized ownership and control are ceded to corporate IT, the CIO’s office, a data governance committee, or some other central organization, sharing customer data across organizational boundaries is relatively fast and easy, as is improving and integrating the managed data.

Most firms have old CDI solutions that need to be updated or replaced.

Of course, the devil’s in the details. Many corporations already have multiple CDI solutions, owned by diverse departments. Many of these are silos connected to a short list of systems, despite their use of integration technologies. And many are legacies that need to be updated or replaced to address new requirements in CDI. In these cases, a corporation cannot start with a clean slate, but must spend time and resources consolidating redundant CDI solutions and updating others. However, the effort of CDI is worthwhile because it yields ROI. The catch is that the ROI is indirect. For instance, up-to-date customer data leads to more efficient customer service, which yields higher customer satisfaction, such that customers churn less. Since CDI is the first link in the chain (and revenue is the last), it’s hard to link CDI to revenue. But the link is there.

CDI has ROI and benefits, so just do it. Most users will update their CDI solutions soon and significantly.

TDWI rese a rch

ROI aside, CDI has other benefits. According to this report’s survey of users, the leading benefits involve improvements to actions in business intelligence (BI), sales and marketing, and customer relations. And improvements in these activities lead indirectly to revenue lift or cost savings. According to our survey, data quality functions are what users wish for most in their CDI solutions. Curiously, the same users admit they aren’t doing much with data quality today, but they anticipate beefing up data quality functions in their CDI solutions in the future. The survey uncovered the same situation with master data management, metadata management, third-party data, bidirectional data movement, and Web services. All are inadequately supported in today’s CDI solutions, but users anticipate correcting all these soon. TDWI hopes the correction does indeed happen—and soon. Without these corrections, CDI solutions are unlikely to satisfy recent requirements (respectively) for consistent definitions of the customer, data lineage, visibility beyond the enterprise, customer data distribution through a CDI hub, and customer data services that are embeddable within packaged and composite applications.

Introduction to Customer Data Integration

For many reasons, it’s difficult to know who your customers are, much less what they’re doing. For example, companies in industries like financial services, insurance, and retail have grown through the organic development of new products and the acquisition of products through mergers. The result is that many firms now have dozens of products and services, which the customer purchases and uses through many touch points, such as call centers, retail outlets, business reply cards, and Web sites. This isn’t just a large-corporation phenomenon; even relatively small businesses have multiple products and touch points. The catch with the complexity of modern business is that it’s difficult to know whether a customer in your small piece of the enterprise is also a customer in other businesses across the enterprise. Without cross-business-unit information, you cannot accurately answer strategic and analytic questions, like: “How profitable is this customer?” or “What other products can we up-sell or cross-sell to this customer?” You may not be able to answer tactical and operational questions, like: “For this product, what are the customer’s shipping and service preferences?” or “What method of payment does the customer prefer?” Far worse, when a customer is identified differently in different businesses, you can’t even be sure you’ve identified the customer correctly when he or she contacts you. Even if you can answer all these questions, you probably have no visibility into the customer’s financial and lifestyle behaviors outside of your enterprise.

Who are your customers? What are they doing? Who knows?!

For the sake of effective customer service, sales, account management, billing, shipping—you name it!—companies need to accurately recognize the customer across multiple lines of business and touch points, plus integrate the customer’s information across those for a more complete picture. To achieve these goals, many companies have turned to customer data integration (CDI).

Definitions of CDI and Related Concepts CDI can take many forms, ranging from simple to complex. But here’s a definition that captures most of its components and goals: Customer data integration (CDI) uses information technologies, business processes, and professional services to collect customer data from disparate enterprise and third-party sources and integrate the data into a singular 360-degree view of each customer that’s complete, up to date, accurate, clean, and standard. TDWI’s position is that customer data is an enterprise asset that should be integrated, shared, and leveraged broadly via CDI techniques. Let’s step back for a moment and define how key terms and concepts are applied in this report. CDI Solution. A CDI solution is a collection of user-designed best practices and vendor-defined functions built into software tools and platforms. The solution’s base requirement is to integrate data from disparate sources. But it must perform other tasks, too, like improve the quality of customer data, reduce redundant customer records, identify and match customers accurately, reconcile conflicting customer references, enrich customer records with data from external third parties, and distribute customer data back out to the IT systems and business units that need it. CDI Hub. A good CDI solution will have an architecture, and hub-and-spoke architecture is the most common one. With simple and legacy CDI solutions, customer data travels inbound spokes into the hub, where it’s persisted in a mid-tier customer database. However, more modern CDI solutions will also serve up improved customer data through outbound spokes, thus making the hub a true bidirectional distribution center. Either way, the integration and reconciliation processing mentioned earlier occurs at the hub.

CDI architecture is usually hub-and-spoke.

w w w.tdwi.org

customer data integration

Customer data services. A major trend with CDI hubs is toward hubs that act as distribution centers. These assume bidirectional travel of customer data in and out of spokes that can embed customer data services in a variety of operational and analytic applications. Customer data services require a range of interface types, including traditional SQL-based access, native gateways, Web services, service-oriented architecture (SOA), and near-real-time connections. CDI has traditionally focused on persisting data, but this is changing.

Customer database. These databases vary a lot. Most homegrown CDI solutions store data coming into the hub in a custom-modeled operational data store (ODS). In basic solutions, the CDI hub is just this ODS and a few data integration routines. If the CDI hub serves business intelligence (BI) purposes, the hub will probably feed a customer data mart, which may be part of the CDI hub or part of an enterprise data warehouse environment. Technical users conceptualize the customer database as a mid-tier database that’s a persistent cache between source and target systems. Federated or virtual CDI. While it’s true that most CDI solutions today relocate and store a lot of customer data at the hub, one trend is taking the hub toward a more federated approach, where as little data as possible is persisted at the hub. In this architecture, customer data is processed on an asneeded basis, typically in small amounts, perhaps just a customer record at a time. This is preferred where customer data changes very frequently or where MDM or SOA is applied.

The point of CDI is to liberate customer data, so it becomes an enterprise asset.

Data as an enterprise asset. TDWI has seen many enterprises run business initiatives called “data as an enterprise asset programs.” In these programs, a central body (like corporate IT or a governance committee) takes ownership of data that has been isolated in departmental silos. The point is to liberate data and share it broadly, so it becomes an enterprise asset. Most of these programs focus on customer data, since its broad sharing has obvious benefits for the enterprise. Hence, CDI is regularly an enabler of a data as an enterprise asset program. TDWI’s position—repeated throughout this report—is that customer data is an organizational asset that should be managed broadly yet carefully via CDI, if you—and your customers—are to get full value from the data. USER STORY CDI can turn customer data into an enterprise asset. Over the years, a chain of retail banks in the American Midwest remained product-centric—focused on loans, mortgages, checking accounts, and retirement funds—while the rest of its industry became more customer-centric. To catch up with industry trends and to regain its competitiveness, the bank resolved to morph into a holistic customer-centric business, driven by customer data as an enterprise asset. The problem with the plan was that customer data was distributed across isolated departmental applications, each owned by a different line-of-business (LOB) manager. Ultimately, the bank succeeded, thanks to a data governance program—directly sponsored by the highest level of management—that focused on business change, with technology kept in a supporting role. Governance got the LOB managers on board, and CDI got customer data flowing across systems and departments.

TDWI rese a rch

Introduction to Customer Data Integration

Analytic CDI and Operational CDI Practices CDI is like many data management practices, in that it can be applied for analytic purposes, operational purposes, or both. To understand the current state of CDI relevant to these two broad practices, TDWI asked survey respondents to identify the purpose of their primary CDI solutions. (See Figure 1.) • Support analytics (26%). Data integration specialists regularly aggregate customer data into data warehouses, customer data hubs, and similar databases in support of analytic applications for customer profitability, churn, fraud, risk, customer base segmentation, and sales pipeline. These analytic applications yield the most accurate and actionable results when operating on large volumes of complete, clean, and standardized customer data, which is why CDI is required.

CDI regularly supports analytic and operational purposes, sometimes via the same solution.

• Integrate operations (17%). One of the most compelling uses of CDI is to integrate disparate business processes by sharing and synchronizing customer data across the IT systems that support the processes. The greatest beneficiaries of this practice are customer-facing departments, like sales, telemarketing, direct marketing, call center, order entry, billing, shipping, and so on. The point of sharing customer data across these departments’ IT systems via CDI is that users in each of these departments get a more complete view of customer activity than they could solely through their department’s/departments’ applications and data. • Both (52%). Both analytic CDI and operational CDI deliver compelling benefits, so it’s not surprising that greater than half of survey respondents report practicing both. Many organizations have multiple CDI solutions, and one reason for this is to satisfy the unique requirements of analytic CDI and operational CDI. In other words, the two practices seem very similar, in that they handle customer data. Yet the two practices require different data models, data transformations, interfaces, approaches to master data management, and speeds of data delivery (as we’ll see in detail, later in this report). For these reasons, analytic and operational CDI are usually enabled by separate technology stacks. • Other (5%). TDWI Research has interviewed users who apply CDI very narrowly within the confines of a single application. Even so, the application may be large, as is the case with applications for customer relationship management (CRM) and enterprise resource planning (ERP). CDI typically integrates into the application customer data from other application brands, application instances, or third-party providers. This certainly isn’t analytic CDI. It’s barely operational CDI in that the sharing of data is one-way into a single application. The purpose of your primary CDI solution is: Support analytics

26%

Integrate operations

17%

Both Other

52% 5%

Figure 1. Based on 337 respondents. Percentages total 100%.

w w w.tdwi.org

customer data integration

Why Care about CDI, and Why Now? There are many reasons why your organization may need to take action now by implementing a new solution for customer data integration or by consolidating and modernizing existing ones. Businesses are still in the process of becoming customer centric. In the early 1990s, many companies—and whole industries, too—shifted from a focus on products to a focus on customers. Product-centric industries like manufacturing and consumer packaged goods suddenly discovered that they have customers, who are the end-consumers beyond the usual supply chain of distributors. Customer-oriented industries like financial services became even more customer centric. As a special case, the insurance industry began to complement its long-standing obsession with policies and similar products with new considerations for customers. All this made the customer even more valuable than ever, which in turn gave data about customers a level of value it hadn’t had before. And the value of customer data increases as data is cleansed, completed, and shared through CDI. Many firms have multiple CRM and/or CDI solutions that need consolidation or modernizing.

Many businesses suffer from multiple, redundant CRM applications. In the late 1990s, as a reaction to the new customer centricity, corporations bought and built droves of applications for various forms of customer relationship management (CRM), as well as applications that include some CRM-ish functions, like applications for billing, shipping, customer service, and enterprise resource planning (ERP). Most of these became silos yet have overlapping functionality. The consolidation of redundant CRM and CRM-ish applications is one of the most common consolidation projects seen this decade, and it’s an attempt to atone for the sinful shopping spree of the late 1990s. CDI regularly assists with the consolidation of CRM applications. However, CDI more often synchronizes customer data across the CRM applications that remain after consolidation. And most CRM applications lack sufficient data integration functions, so CDI adds a necessary function to these. As with CRM applications, CDI solutions are numerous and redundant in some businesses. TDWI Research interviewed users who work in enterprises that have a dozen or more CDI solutions. They have one each for marketing, sales, financials, operations, customer services, and so on. Some even have multiple analytic CDI solutions, one each for customer base segmentation, churn analysis, fraud detection, sales pipeline analysis, and customer profitability. In some cases, these sprang up to correct the insufficient data integration of CRM and ERP applications. In other cases, these CDI solutions are spreadmarts, hacked together by power users to serve a single department. Regardless of their origins, redundant CDI solutions proliferate multiple views of the customer—the very illness CDI is meant to cure! They also drive up IT maintenance costs and threaten compliance violations. Hence, many businesses need to gain control of customer data usage by consolidating redundant CDI solutions.

Many CDI solutions are silos or legacies that need a replacement.

TDWI rese a rch

CDI is often a bunch of dots that need connecting. CDI is practiced in multiple solutions per enterprise, where each is a kind of silo, and there are holes between silos. In this regard, CDI is in a state similar to that of other data management techniques, such as data integration, data quality, and master data management, which are evolving from a point solution practice to an enterprisewide practice. Connecting the dots of CDI silos is more than a consolidation project. Many businesses need to rethink CDI entirely to make it vibrant and relevant for the entire enterprise, not just isolated departments.

Introduction to Customer Data Integration

Believe it or not, most CDI solutions are legacies. To make the CDI proliferation problem worse, many CDI solutions were acquired or homegrown in the 1990s, so today they’re legacies in need of replacement by an up-to-date implementation that addresses the organization’s current requirements. Furthermore, legacy CDI solutions usually lack modern CDI functionality, like data quality, master data management, Web services, and real-time interoperability. Without this functionality, a CDI solution can never reach across an entire enterprise or be an effective distribution hub. CDI techniques can be adapted to data domains beyond the customer. As we’ve seen, CDI is a hot topic in industries that are recognizably consumer oriented, such as financial services, insurance, retail, and so on. But in other industries, it can be adapted to business entities that may be considered equivalent to the customer, like the patient in healthcare, the citizen in government, and the business partner in business-to-business (B2B) oriented industries like manufacturing. CDI is a technology that you should never deploy for its own sake. Instead, the integration and sharing of customer data should support grander goals that affect the business’s bottom line or revenue potential. For example, in firms that rely heavily on direct marketing, CDI can both reduce the costs of direct mail and improve its conversion rates. In industries that suffer high rates of customer churn—say, financial services, insurance, or telecommunications—CDI can contribute to analytic insights about customers’ behavior prior to churn and thus help you react quickly enough to retain more customers.

CDI has ROI when it supports applications or business processes that lift revenue or reduce costs.

Business Issues Concerning the Use of Customer Data Before applying integrated customer data to certain business activities, consider that customer data is subject to a number of regulations, customer relationship taboos, organizational dynamics, and indirect ROI. Regulations. Customer data models a very special business entity—the customer. Customers are special because they are human beings, citizens, and a source of revenue. For these reasons, a number of regulations (both internal and external) protect them. For example, legislated laws—like HIPAA and Basel II—limit how you can legally integrate and distribute data about people. Other laws and corporate policies define the privacy and security of customer data and thereby control how customer data can be used. Be sure to check with your legal counsel and compliance officer before initiating new uses of customer data.

Customer data usage must comply with external regulations and internal policies.

Customer relations. At one extreme, some customers become indignant when they realize your call center representative has no idea who they are or why they’re calling. At the other extreme, some customers get paranoid when they realize you know who they are and where they live. Customer satisfaction is a tactical goal and an absolute requirement for most CDI and CRM initiatives, and it requires a balance between your zeal for collecting and leveraging customer data and your discretion in how and how much data is revealed to the customer. Organizational dynamics. As we’ll see later, the most formidable barriers to CDI are cross-functional organizational issues and data ownership, hurdles that are best vaulted via data governance. These and similar barriers are even more daunting in the context of business initiatives known as “data as an enterprise asset” programs. These programs are highly disruptive, in that they take data ownership away from the business units and departments that funded them (and typically depend on them daily) in favor of centralized data ownership, control, and management. Though radical, data as an enterprise asset is necessary in strongly siloed organizations that assiduously resist the sharing of data and other resources. Data as an enterprise asset programs typically involve some form of CDI

CDI is cross-functional, so it benefits from data governance.

w w w.tdwi.org

customer data integration

because customer data is a highly valued asset. In fact, most of these programs focus on customer data primarily, and the integration of customer data is an early phase deliverable. Hence, all CDI programs have organizational dynamics to cope with, but CDI in the context of a data asset program faces the fiercest challenges. CDI’s ROI is tough to see, but it’s there.

Indirect ROI. The business case for CDI is both easy and hard to make. Note that most CDI scenarios include an implicit domino effect. For instance, up-to-date customer information leads to more efficient customer service, which yields higher customer satisfaction so that customers churn less. Or more complete customer data enables more accurate customer segmentation and one-to-one marketing, which leads to better targeted marketing campaigns, resulting in higher conversion rates. Since CDI is the first link in each of these chains—and the effect on revenue is the last—it’s hard to link a direct causal effect from CDI to revenue. We sometimes forget the soft benefits of the intervening dominos. Even so, conventional wisdom, based on the hindsight of many CDI initiatives completed, now says that CDI easily yields a return on the investment, whether raising revenue through sales and marketing or retaining customers through better service. Another way to put it is that CDI is an underlying data management infrastructure that contributes to business initiatives and their financial goals, such that CDI yields ROI, albeit indirectly. USER STORY CDI contributes to database marketing success. “My job in direct marketing is all about increasing retention rates and customer’s share of wallet,” said Ivona Piper, the former VP of database marketing at Staples, Inc., the world’s largest office products company, with 2,000 stores in 22 countries. “At Staples, we believe that conversion rates are accelerated by more complete and precise customer data, which in turn yields more accurate customer segmentation and other insights. Our primary goal was to identify our ‘best customers’—as seen in high spending or repeat business—so we could focus on them. The problem is that even the best customer looks small when seen from only one touch point or data silo.

“So we embarked on a CDI project that consolidated 19 customer databases into one. The resulting customer data hub reconciles customers across multiple programs and touch points, and it groups customers who work for the same company. In the first year of use, response rates for e-mail promotions and other campaigns fed from the newly integrated customer database rose from single-digit percentages to double-digit ones. The CDI project cost more than we thought it would, but this was easily offset by reductions in the number of systems and headcount in both IT and marketing.”

TDWI rese a rch

The State of CDI

The State of CDI The Perceptions and Realities of Sharing Customer Data The first question in this report’s survey asked respondents to rate their organization’s perceptions and efforts as high, medium, or low for issues related to sharing customer data. (See Figure 2.) In the question, issues were grouped in pairs to reveal conflicts between perception and reality. • Most organizations think that sharing customer data is highly valuable (59%). But sharing mostly reaches medium and low percentages of the enterprise (44% and 38%, respectively). Here, as with many CDI issues we’ll see in this report, the perception of potential benefit is way ahead of the action being taken to achieve the benefit.

On the upside, users think that CDI is valuable, they should have access to it, and it yields benefits.

• Employee access to customer data is medium to high in most organizations. Yet, the completeness of the data is mostly low (55%). The good news is that organizations are sharing customer data; the bad news is that the data being shared is sketchy. • All respondents rated very highly both benefits and problems. In fact, few respondents rated benefits or problems as low. Clearly, shared customer data yields perceptible benefits, just as the lack of it results in noticeable problems. • The perceived quality of customer data is mediocre, as is the effort put into improving it. This isn’t bad, given that customer data (due to its constantly changing nature) is more prone to quality problems than most data domains. This explains why the majority of data quality solutions focus on customer data, whereas other domains get little or no quality improvement. • Half of respondents consider their organization’s CDI success mediocre (50%). Few rate their success as high (11%), and a considerable percentage (38%) rate their success as low. This shows that CDI solutions have plenty of room for improvement in most organizations.

On the downside, users don’t invest enough in CDI, customer data is incomplete, and CDI success is mediocre.

Rate your overall organization for: HIGH

MEDIUM

Perceived value of sharing customer data

59%

Percent of enterprise sharing customer data

18%

Employee access to customer data

19%

Completeness of single views of customers

15%

51%

Overall level of success with CDI

11%

55% 30%

20% 39%

10% 42%

47% 29%

11%

36%

30%

Problems suffered from a lack of sharing

10% 38%

45%

50%

Current effort for improving customer data

31% 44%

Benefits gained from sharing customer data

Perceived state of customer data quality

LOW

47% 50%

24% 38%

Figure 2. Based on 352–357 respondents per answer.

w w w.tdwi.org

customer data integration

The Scope of CDI Solutions Half of users claim CDI is already reaching their entire enterprise.

One of TDWI’s positions in this report is that organizations don’t share enough customer data or share it broadly enough. In a related trend, CDI is like a lot of data management practices in that it’s broadening beyond departmental silos into enterprisewide usage. Hence, the scope of individual CDI solutions—plus the aggregate effect of multiple solutions—is an important metric for gauging the breadth of customer data sharing. In an effort to quantify the current state of CDI scope, TDWI Research asked survey respondents: “For your primary CDI solution, what is the scope of customer data sharing?” (See Figure 3.) Half of respondents (49%) claim that their organizations are already practicing CDI with enterprise scope, while another 14% report reaching the enterprise from a departmental CDI solution. This is a surprisingly high number of organizations reaching the enterprise, given that CDI has a reputation for departmental and spreadmart solutions. Users interviewed for this research reported similar progress, corroborating the survey data. Also, many interviewees described work they had done in recent years to consolidate CDI silos into fewer solutions, extending the reach of the surviving solutions in the process. The gist of the market data is that many organizations are well down the road to enterprise-scope use of CDI techniques and practices—and that’s a big improvement for CDI. For your primary CDI solution, what is the scope of customer data sharing? The enterprise

49%

A business unit A department (with enterprise reach) A department (isolated)

26% 14% 11%

Figure 3. Based on 344 respondents. Percentages total 100%.

The Number of CDI Solutions per Organization TDWI’s impression is that the number of CDI silos and other isolated customer data solutions has decreased in recent years. After all, many corporations have run consolidation programs to reduce the number of databases and data marts—and many of these are customer data hubs. And don’t forget: customer data hubs have been around for more than 15 years, so CDI solutions deployed in the 1990s have, this decade, become legacies needing migration or retirement. To get a feel for the number of CDI solutions per organization today, TDWI Research asked survey respondents to “enter the number of CDI solutions in your enterprise.” (See Figure 4.) Most organizations have one to five CDI solutions.

• Organizations have 5.2 CDI solutions on average. This is a manageable number, and it’s less than TDWI Research anticipated. In fact, multiple solutions are desirable in some companies, especially multinational firms with distinct customer bases that are complementary by region. • Some organizations have no CDI solution, at all. Although CDI has been with us for decades, there is still a minority of firms (13%) that aren’t leveraging and sharing customer data officially through a CDI solution. • The majority of organizations (60%) have one to five solutions. Again, this is good news, showing that most organizations have successfully controlled the proliferation of CDI solutions.

TDWI rese a rch

The State of CDI

• A few respondents (18%) have 6 to 10 CDI solutions. If you have more than five, you might consider tightening controls on the spread of customer data. • A minority of respondents (7%) has more than 10 CDI solutions. If you fall into this category, your situation is highly abnormal and needs immediate attention, perhaps via a consolidation project.

Some organizations have 100 CDI solutions, and others have none.

• Some organizations have 100 or more CDI solutions. Only 2% of survey respondents fall into this category. An extreme case like this is reached when an enterprise doesn’t control the proliferation of customer data marts, spreadmarts, and operational data stores. Hence, great numbers of CDI solutions are possible, but not common—luckily! Enter the number of CDI solutions in your enterprise. Zero

13%

1 to 5

60%

6 to 10 Between 10 and 100 100 or more

18% 7% 2%

Figure 4. Based on 308 respondents. Percentages total 100%.

USER STORY CRM consolidation is a common penance for the sins of the 1990s. A midsize software vendor in New England went on a shopping spree in the late 1990s—as did many companies—and bought and built several CRM and CRM-ish applications for internal use. Most of these applications overlap, because they automate customer-oriented functions such as order entry, shipping, billing, direct mail, customer base analysis, and multiple approaches to sales force automation. Synchronizing customer data across these is too complex to be effective, so the firm does not have a clean or complete view of customers. Today, this firm is paying for the indulgence of the 1990s by migrating and consolidating redundant applications and their databases into a single instance of a packaged application for CRM, plus extending that instance to fill in missing functions. This is a multiyear, multistage project that employs many data management tools, as well as extensive process reengineering to ensure that end users and the business get full value from the new application.

CDI Funding, Use, and Maintenance It’s always telling to discover which departments fund, use, and maintain an information system, since this reveals who has the money versus who has the greatest need for the system. To sort this out, this report’s online survey instructed respondents to enter the name of the persons or departments that fund CDI, need it the most, and maintain it. There were no prewritten answers to select; instead, each of the 294 respondents typed an answer in his or her own words. The answers were quite diverse, but a handful of departments and job titles appeared much more commonly than others:

IT gets stuck funding and maintaining CDI more than any other department.

• Information technology (IT) is the most common source of CDI funding (according to 24% of survey respondents), and IT usually gets stiffed maintaining it (63%). Yet, IT rarely needs or uses CDI (3%). • Marketing and sales occasionally fund CDI (19%), and they are certainly its most common users and grateful beneficiaries (38%). But they rarely lend a hand to maintain it (7%).

Marketing and sales are the most common users of CDI solutions. w w w.tdwi.org

cus t omer data in t egr at ion

• Corporations and enterprises (12%) and chief officers of various types (10%) were listed as funding sources, though none of them use or maintain CDI. • Business intelligence (BI) teams weren’t identified as a source of funding for CDI, although they occasionally use (3%) and maintain (6%) it.1

Benefits of CDI The Internet-based survey for this report posed a question that listed known benefits of CDI, and it asked survey respondents to rate the impact of each potential benefit as high, medium, or low. From their responses, we can judge the likelihood of these benefits, as shown in Figure 5. Rate the impact of benefits from CDI. (Select one answer per row.) HIGH

MEDIUM

Enhanced customer analyses

LOW

72%

Achieved 360-degree customer view

21%

64%

25%

11%

Enlightened strategic decisions

61%

30%

Improved customer service

61%

29%

10%

Revealed customer opportunities

57%

Enabled cross-sell and up-sell

51%

Made data an enterprise asset

51%

Raised customer satisfaction

32% 29%

36%

Reduced customer churn

34%

12%

31%

39%

Supported risk and compliance efforts

14%

37%

45%

Reduced operational costs

20%

35%

51%

Targeted direct marketing better

11%

24%

40%

21%

40%

24%

37%

Improved billing and collections

27%

33%

Increased conversion rates

26%

36%

29% 40% 38%

Figure 5. Based on 318–333 respondents per answer.

Conventional wisdom tells us that sharing clean, complete, and integrated customer data yields benefits in three broad areas: business intelligence (BI), customer relationships, and sales and marketing activities. It’s no coincidence, then, that these three areas ranked well in the responses of survey respondents: Most CDI benefits relate to BI, sales, or marketing.

BI-related goals topped the list of benefits. The most likely benefits of CDI (according to the survey) all concern BI goals, like “enhanced customer analyses,” “achieved 360-degree customer view,” and “enlightened strategic decisions.” Clearly, among survey respondents, the most compelling reason for implementing CDI is to achieve BI goals. Sales and marketing activities gain high impact through CDI. In particular, these include “revealed customer opportunities,” “enabled cross-sell and up-sell,” and “targeted direct marketing better.”

TDWI rese a rch

1 According to these numbers, BI teams don’t use CDI much, which is inconsistent with other findings of this report. For example,

in Figure 9, data warehouses and customer analytics are the most common applications integrated with a CDI solution.

The State of CDI

Oddly enough, the related issue of “increased conversion rates” ranked low. All these activities lead directly to the ultimate benefit—a lift in revenue—which by extension indicates that investments in CDI lead to a return. Customer relationship benefits are all over the map. For instance, “improved customer service” ranked very high, “raised customer satisfaction” ranked roughly in the middle, and “reduced customer churn” ranked rather low. As a user interviewed by TDWI Research for this report put it, “In prioritizing departmental support for our CDI implementation, customer relationship activities that yield socalled soft benefits tend to take a back seat to revenue-producing sales and marketing activities.” Some issues are not likely benefits for CDI. These include “reduced operational costs,” “supported risk and compliance efforts,” and “improved billing and collections.”

Barriers to CDI Despite potential benefits, survey respondents also perceive several barriers to CDI. (See Figure 6.) Their responses rate the severity of each potential barrier as high, medium, or low. Territorial disputes are the leading barrier to CDI. More respondents gave a high severity rating to “data ownership and other territorial issues” and “lack of cross-business-unit coordination” than any other barriers listed in the survey. These issues are so serious that they can halt CDI in its tracks. After all, sharing customer data assumes a fair amount of collaboration among business managers and other owners of IT systems. Knowledge is power and customer data is valuable, so occasionally a manager or system owner declines to share. In a related issue, “lack of governance” also ranked as a severe barrier. Various forms of governance—especially data governance—can provide the cross-functional coordination that expansive data management practices like CDI demand.

CDI’s severest barriers are ownership and turf issues.

Cost is a concern with CDI. But, then, cost is an issue with all technology solutions, and the survey ran during an economic downturn. Although “cost of CDI” ranked high, “quantifying return on investment” was not considered as severe a barrier. Technology can be a barrier to CDI. To state the obvious, CDI is a data integration practice, so an “inadequate integration infrastructure” can be a problem, especially when a CDI solution must reach a long list of source and target systems. A similar barrier—“incomplete customer data”—reminds us that it’s hard to integrate customer data when customer data isn’t currently captured fully by IT systems or fully shared by system owners. Furthermore, data quality has traditionally been a challenge to CDI; but this issue seems to have improved in recent years, as shown by the low severity given to “low quality of data.”

CDI may be inhibited by incomplete data or inadequate integration infrastructure.

Getting and keeping executive sponsorship can be a problem. The potential barriers of “lack of executive sponsorship” and “nonsustainable executive sponsorship” received medium-to-low severity in the survey. An influential sponsor is key to gaining funding for CDI, as well as for convincing diverse departments to contribute data and other resources. Some issues are not likely barriers to CDI. These include “data privacy or security issues,” “modeling integrated customer data,” and “compliance and regulations.”

w w w.tdwi.org

customer data integration

Rate the severity of barriers to CDI. (Select one answer per row.) HIGH

MEDIUM

Data ownership and other territorial issues

LOW

61%

Lack of cross-business-unit coordination Cost of CDI Lack of governance

48% 45%

Incomplete customer data

45%

Lack of executive sponsorship

43%

Quantifying return on investment

42%

14%

36%

53%

Inadequate integration infrastructure

14%

25% 30%

56%

11%

37%

15% 17%

38% 41%

14%

32%

25%

40%

18%

Resistance to change or transformation

41%

18%

Data privacy or security issues

39%

43%

18%

47%

17%

Low quality of data

36%

Nonsustainable executive sponsorship Modeling integrated customer data Compliance and regulations

33%

41%

25% 18%

43% 45%

26% 32% 37%

Figure 6. Based on 322–332 respondents per answer.

USER STORY CDI is hamstrung if customer contact data isn’t captured online. “We have excellent data integration and quality assurance measures for the aggregated data we sell, but we’ve only recently started applying these techniques to customer data internally,” said Ron Nagel, a senior associate in the Information Product Services division of Mercer LLC, a global leader in products and services for human resources. “For the information products my division of Mercer sells, we have few contacts with a customer per year, yet each transaction has a fairly high dollar value. So we have to make the most of each contact, to give the customer the best service and to seize opportunities for cross-sell and up-sell.”

Mercer Information Product Services is addressing this challenge through a new CDI project. “CDI’s great for data that exists in our systems. But we found that we needed to capture more data about each contact, so we’d have more data to work with after the contact,” Ron explains. “We altered our sales processes to capture information about the channel through which the contact was initiated, why the customer is buying from us (instead of a competitor), the customer’s unique identification (so we don’t create a new, redundant customer record), and so on. Now, we’re getting the data we need to leverage every contact in near real time and to study our customer base as a whole, offline.”

Key Features of a CDI Solution

CDI solutions are very diverse when it comes to the software functions built into them. There’s a broad range of possible functionality, involving different forms of data integration, data quality, interfaces, databases, and semantic data management. And all this can be enabled by a range of applications and tools, as well as hand coding.

TDWI rese a rch

Key Features of a CDI Solution

To get a sense of users’ priorities among diverse CDI features and functions, this report’s survey asked users to “rate the priority of various features for a CDI solution.” (See Figure 7.) The multiple choices of this question constitute a long list, because of the diversity of CDI solution functionality. Survey respondents rated the priority of each feature as high, medium, or low, and Figure 7 sorts features according to the percentage of respondents who rated the feature as high priority. This sort order suggests a priority order among the features. Before looking at their ratings, let’s realize that survey responses represent users’ opinions, and the opinions are sometimes based on actual experience, and sometimes on what users would prefer to do if they had the right resources and opportunities. Either way, the features that bubbled to the top of the priority order seen in Figure 7 are good indications of what works and even better indications of users’ strongest requirements. Readers should compare the features listed in Figure 7 to features of their own CDI solutions. This helps organizations assess the current state of such solutions and plan which features to add in the future. The list could also aid in the selection of vendor products and consulting services for CDI. Rate the priority of various features for a CDI solution: HIGH

High-Priority CDI Features

MEDIUM

Data quality functions Built-in data integration or ETL

Low-Priority CDI Features

24%

58%

Standardization for better linking and matching

Medium-Priority CDI Features

LOW

70%

55%

30%

12%

35%

10%

Batch bulk load of data

54%

33%

13%

High accuracy matching and “survivalship”

53%

33%

14%

Master data management

46%

Bidirectional data synchronization

45%

Scale to big data sets, high-speed access

43%

37%

17%

31%

24%

38%

19%

Data load and access in real time

41%

Workflow to approve data changes

41%

Federated access to customer data

40%

41%

19%

Persistence for detailed source data

40%

42%

18%

Self-service search and discovery

40%

Tool for data stewards and governors

39%

Service-oriented architecture (SOA)

37%

High volume match in real time Prebuilt customizable data model

29% 27%

Third-party data for enrichment

27%

Native interfaces to packaged applications

25%

39% 34%

20% 25%

41%

19%

38%

23%

39% 37% 38% 42% 43%

24% 34% 35% 31% 32%

Figure 7. Based on 324–339 respondents per answer.

w w w.tdwi.org

customer data integration

The CDI solution features, as sorted in Figure 7, break into three rough groups for high-, medium-, and low-priority CDI features:

High-Priority CDI Features Data quality and data integration are the highest priorities for CDI solution features.

In users’ minds, the highest priority features all relate to data quality and data integration. In fact, “data quality functions” was rated a high priority by an overwhelming 70% of survey respondents, greater than for any other CDI feature. “Built-in data integration or ETL” (58%) occupies a somewhat distant second place. The priority of data quality and integration isn’t surprising. After all, CDI is a domain-specific form of data integration, and we all know that customer data (due to its highly changeable nature) is systemically susceptible to data quality problems and opportunities. Another way to think of it is that data quality (DQ) and data integration (DI) functions are base requirements, and successful CDI isn’t possible without them. On the upside, users interviewed by TDWI consistently pointed to DQ and DI as key success factors. On the downside, they are also the lion’s share of development time and cost, whether CDI solutions are hand-coded, built atop a vendor’s tool, or based on a dedicated CDI or MDM application. Data quality’s priority is even higher when we consider that many features rated high in the survey are usually accomplished with data quality tools or techniques. In particular, third and fifth places in the survey were taken respectively by “standardization for better linking and matching” (55%) and “high accuracy matching and ‘survivalship’” (53%). Standardization is the leading action taken by a DQ tool in any solution (as proved by TDWI’s DQ report surveys). With CDI solutions, standardization assures consistency across customer data drawn from multiple systems. It’s also a prerequisite for one of the most important DQ features for CDI—matching. One of the points of CDI is to reduce customer data redundancy. Hence, as customer records from diverse sources are integrated, it’s critical to match records that refer to the same customer. Once matches are made, the CDI solution should automatically perform survivalship, which is usually a rule-driven method for selecting the best record, into which matched records are merged. Various data integration features also fared well in the survey. For example, “batch bulk load of data” (54%) came in fourth place, reminding us that most CDI solutions (especially home-grown and legacy ones) move large volumes of data and rely on batch-driven routines for most data processing. But this high priority is complemented by medium-priority DI features, namely “data load and access in real time” (41%) and “federated access to customer data” (40%). This shows that CDI (like DI for data warehousing) relies mostly on batch DI, because the integration and remediation of most customer data is not time sensitive. Yet, CDI is branching out to embrace real-time and federated approaches to DI for time-sensitive customer data and customer touch points.

Medium-Priority CDI Features MDM, two-way synch, and self service are medium priorities.

TDWI rese a rch

Master data management (MDM) is currently a medium priority for CDI (46%), but on the cusp of becoming a high priority. Don’t forget that most CDI solutions predate the current practice of MDM, so most don’t include MDM features. In fact, multiple people interviewed for this report pointed to MDM as the leading deficiency in their CDI solution. “Incorporating MDM is the best reason that I can think of for updating our CDI solution,” said one interviewee. “If I had the resources, I’d expand our MDM infrastructure, then build CDI solutions atop it.” Considering that MDM is still a relatively new practice, it’s impressive that MDM rated as highly as it did in the

Key Features of a CDI Solution

survey. Anyone designing a new CDI solution or updating an old one should put MDM near the top of the wish list, because MDM provides one of the key requirements of CDI: consensus-driven definitions of the customer, applied consistently across multiple IT systems.2 Another upcoming CDI feature is “bidirectional data synchronization” (45%). Let’s recall that many solutions move data in a one-way flow from source systems to a customer database in a data warehouse, data mart, operational data store, or some other database. This is fine for analytic CDI and some direct marketing applications. But other applications require a two-way flow of data. Common examples include sharing data across multiple customer-facing applications, updating systems with revised master data describing customers, and many data quality functions. One of the biggest changes in CDI over the years is the evolution of the CDI hub from being just a destination to also being a distribution point for cleansed and completed customer data. In recent years, search technologies have been incorporated into more and more enterprise applications, to give users greater ease of use in finding information. This is relevant to CDI, which benefits from “self-service search and discovery” (40%). Two scenarios stand out. First, search is one of the many tools a data steward or similar user needs to discover and understand available customer data. Second, corporate workers in customer-facing jobs (like customer service, financials, and sales) benefit from a mechanism for finding more information about a specific customer than is readily available from a traditional application user interface. In these cases, customer data integrated via CDI is also indexed by a search crawler, and the search index is accessible from a corporate portal, any browser, or (in some cases) an application’s user interface. At the bottom of the mid-priority CDI features is “persistence for detailed source data” (40%). Believe it or not, it’s a good thing that this feature has fallen to a low priority. With any luck it will fall further, and here’s why it should. Recall that almost all early CDI solutions followed a data warehousing architecture, where customer data was extracted and loaded into a database of persistent customer data. Over time this preference has given way to more modern approaches that are bidirectional, federated, virtual, near real time, and based on services. Persistent data stores built via CDI still make sense for some analytic CDI and direct marketing applications. But persistence is a barrier to many of the more modern requirements that have arisen for CDI hubs.

Low-Priority CDI Features Service-oriented architecture (SOA) was rated as a low priority in TDWI’s CDI survey. Yet—like MDM—SOA is a feature that interviewees regretted not having in their current CDI solution, because they know they will need it in the next few years. One interviewee lamented that “my employer’s commitment to composite applications demands that many of my team’s business intelligence, data quality, and customer data integration functions be exposed as services by 2010.” As architectures for CDI hubs continue to evolve, they move more to approaches for federated and virtual access to customer data that can interoperate in near real time and be embedded in a range of application types. Furthermore, CDI hubs are becoming better distribution mechanisms, by supporting bidirectional synchronization of customer data and master data. The ideal interface for all these forward-looking requirements is the service, whether a simple Web service or a full-blown SOA. Again, SOA is somewhat ignored at the moment, but it will soon be a common requirement for CDI.

SOA is ignored today, but will soon be a CDI priority.

Only 27% of survey respondents rated “third-party data for enrichment” as a high priority, which makes it seem like a low priority. But a sizable 42% rated it as a medium priority, giving it more clout. “It’s myopic to just look at your own company’s customer data,” said an interviewee. “If you 2 For a definition and a discussion of MDM, see the TDWI Best Practices Report Master Data Management: Consensus-Driven Data Definitions for

Cross-Application Consistency, available online at www.tdwi.org/research/reportseries.

w w w.tdwi.org

customer data integration

want to know what your customers are doing in the broader marketplace, you need to purchase third-party data and append it to the customer records of your CDI and CRM systems.” Another interviewee pointed out that their CDI solutions have integrated third-party customer data for almost 20 years. So it’s a tried-and-true practice many users take for granted, especially in retail and consumer packaged goods. Despite the hegemony of DQ features, some of them rated low. For example, “high volume match in real time” (29%) is a low-priority feature, probably because real-time operation with large data volumes isn’t a pressing requirement for most CDI solutions (although matching is). Likewise, “tool for data stewards and governors” (39%) was rated low, although tools of this type are built into most modern DQ tools and some DI tools. One of the benefits of a packaged CDI application from a vendor is its “pre-built customizable data model” (27%), which survey respondents felt is a low priority. Likewise, respondents weren’t excited about “native interfaces to packaged applications” (25%), although most CDI solutions interoperate with packaged applications for CRM, marketing campaign management, sales contact management, and financials. EXPERT COMMENT Customer data flows both into and out of a modern CDI hub. “There are really two separate definitions of the term customer data integration,” according to Jill Dyché and Evan Levy, who wrote the book on CDI—literally.3 “On its surface, the term refers to a set of tried-andtrue methods and technologies for bringing customer data together from different data sources, a classic problem for both operational and reporting applications. But the emerging definition of CDI is more specific to the evolving technological capabilities that automate the reconciliation and synchronization of the data from disparate systems in order to offer it to the enterprise for a range of uses and processing.”

The ramification is that a truly modern CDI hub must support both of these definitions, as Dyché and Levy explain: “The CDI hub is the core of a real-time, customer data provisioning environment. The hub is basically the point of recognition of processing for assembly and storage, functioning as the central repository for both the incoming, fragmented data as well as the outgoing, authoritative records.” A real-time, bidirectional CDI hub of this sort will provide customer data services across a broad infrastructure. “CDI essentially functions as a service to other systems; thus, SOA is a critical component of any CDI framework. Web services are the means by which data is captured, synchronized, and shared between CDI and other systems.”

TDWI rese a rch

3 See Jill Dyché and Evan Levy, Customer Data Integration: Reaching a Single Version of the Truth, John Wiley & Sons, 2006, pages 37, 56, and 57.

For a taxonomy of possible CDI hub architectures, see Dyché and Levy, page 183f.

Key Features of a CDI Solution

Buy versus Build “Build a custom solution” is the resounding answer to the buy-versus-build question relevant to CDI. (See Figure 8.) Almost half of survey respondents reported that they “built a custom solution using mostly internal resources” (46%), while an additional 19% reported that they “engaged consultants to build a custom solution.” Hence, a combined 65% of CDI solutions are custom-built by IT or consultants. By comparison, few respondents reported acquiring a vendor’s packaged solution, whether they “customized it lightly” (17%) or “customized it heavily” (12%).

65% of CDI solutions are custom built.

Almost half of the write-in responses to the response “other” (6%) involve some form of outsourcing. This includes outsourcing to offshore development shops, software as a service, vendors’ hosted applications (usually for CRM or sales automation), and boutique shops for database marketing or data quality. And a few respondents wrote in “all the above,” reminding us that some organizations have multiple CDI solutions from multiple origins. For your primary CDI solution, what was your approach to implementation? Built a custom solution using mostly internal resources

46%

Engaged consultants to build a custom solution

19%

Acquired a vendor’s packaged solution and customized it lightly

17%

Acquired a vendor’s packaged solution and customized it heavily Other

12% 6%

Figure 8. Based on 323 respondents. Percentages total 100%.

Applications Integrated with CDI Solutions TDWI asked survey respondents to identify the types of applications integrated with their primary CDI solution, in an effort to develop a list of source and target systems for CDI. The responses are predictable, resulting in a ranking that highlights integration with the systems most people would associate with CDI, namely (in priority order) applications for business intelligence, sales and marketing, and customer-facing operational applications. (See Figure 9.) • The applications most commonly integrated with CDI involve business intelligence. Topping the list for integration today are data warehouses (58%) and customer analytics (53%). Very few organizations have “no plans” to integrate with these.

CDI is integrated with BI more often than with other systems.

• Sales and marketing applications are also commonly integrated with CDI. These include applications for CRM (42%), marketing campaigns (40%), and sales contacts (34%). In addition, at least 40% of respondents plan to integrate with each of these in the future. • Customer-facing operational applications are lesser priorities. These include applications for financials (34%), order entry (33%), billing (32%), and call center (31%). • Some applications have minimal relevance to CDI. This includes applications for ERP, telemarketing, and supply chain management. For each of these, at least 40% of respondents have “no plans” to integrate with a CDI solution.

w w w.tdwi.org

customer data integration

For your primary CDI solution, identify the types of applications integrated with it: NOW

FUTURE

Data warehouse

NO PLANS

58%

Customer analytics

32%

53%

CRM

42%

Marketing campaign management

34%

Financials

34%

42%

40%

Sales contact management

10%

39%

16%

42%

18%

40%

26%

34%

32%

Order entry and management

33%

24%

43%

Billing

32%

27%

41%

Call center

31%

ERP Telemarketing Supply chain management

35%

28% 18% 14%

34%

32% 31% 31%

40% 51% 55%

Figure 9. Based on 314–335 respondents per answer. On average, CDI is fed by 10.3 internal sources and 3.3 external ones.

This report’s survey also asked respondents to enter the number of systems that feed data into their primary CDI solution, as well as the number fed from it. (See Figure 10.) • More systems feed into CDI than out of it, for both internal and external systems. As pointed out earlier, with most CDI solutions, customer data flows one-way into a database at the hub, but little data flows out again. This will change in upcoming years, as more organizations add bidirectional data synchronization to their CDI hubs. • CDI integrates with far more internal systems than external ones. In fact, 33% of survey respondents integrate with zero external sources, and 55% integrate with zero external targets. Even so, 66% of respondents integrate with at least one external source, possibly to acquire third-party customer data, which is a common practice in customer-oriented industries. • The relatively low number of sources and targets suggests low complexity. This makes CDI downright simple compared to similar practices like ETL for data warehousing. It also suggests that CDI solutions have room to grow, in terms of adding new sources and targets or greater data volumes and processing functions. For your primary CDI solution, enter the number of: Average Internal systems that feed data into it

10.3

Internal systems that are fed from it

7.6

External systems that feed data into it

3.3

External systems that are fed from it

2.4

Figure 10. Based on 301 respondents.

TDWI rese a rch

Vendor Tools and Platforms

USER STORY Begin with a short-term goal; plan for a long-term enterprise solution. “For years, we’ve been developing new, data-delivery products,” said Becky Briggs, the senior manager and data steward at ARC (Airlines Reporting Corporation). “For example, our Travel Industry Data Warehouse (TIDW) is a multi-terabyte data warehouse environment that manages over three years of detailed and summary data about ticket purchases. A wide variety of customers—such as airlines, airports, and industry analysts—purchase products created utilizing the TIDW, so they can understand their customers’ behavior in a broader industry context.

“The data that makes the TIDW possible comes from our internal systems and external sources,” Briggs explained. “ARC handles most transaction processing for all travel agency-sold airline tickets in the United States. And we have direct contact with airlines, airports, travel agencies, and other customers through various products and services. The catch is that customer data is strewn across multiple business units and IT systems. We’ve known for years that we should integrate and improve customer data, for the sake of internal efficiencies and external data products. “So, a year ago we started an enterprisewide project,” Briggs continued. “The first phase focused on basic product and customer data integration (CDI), because we had a short-term need for revenue lift based on customer analysis. This was a good decision, given the tough sales year 2008 has become. The phase-one CDI work succeeded, and it’s now a foundation for the second phase of implementing a new packaged application for enterprise CRM and consolidating some of our redundant applications into it. The resulting CDI and CRM solutions will change the face of how we do business, and we need this change to serve our customers better and to adapt to an evolving business world.”

Vendor Tools and Platforms that Enable CDI

As we saw earlier, there are many technologies available that can contribute to CDI solutions. And users definitely have strong opinions about which of these they would prefer to use, given sufficient resources and opportunities. The priority rankings of Figure 7 are useful for planning new CDI solutions, at least in an ideal world. But now it’s time to determine which of the available technologies users are applying to CDI now, in the real world, and which they plan to add on in the future. As we’ll see, users’ priorities (which tend to be future oriented) differ markedly from what they’re actually doing today. In this report’s user interviews, certain types of data management tools came to the foreground as credible enablers for CDI solutions. To determine which of these are truly being used, this report’s online survey presented a list of tool types and asked users to identify which they are using now, plan to use in the future, or have no plans to use. Figure 11 sorts the list of tool types according to which are used most now. For example, the resulting sort order reveals that database management systems are the most common technology applied to CDI, whereas vendor tools for CDI or MDM are the least used today.

w w w.tdwi.org

customer data integration

For your primary CDI solution, identify the tool and technology types you use: NOW

FUTURE

Database management systems

82%

Data integration or ETL

12%

73%

BI reporting and analysis tools

66%

Data warehouses and marts

66%

Data cleansing and profiling

41%

Third-party customer data

41%

Metadata management Vendor tools for CDI or MDM

23%

28%

26%

45%

11%

41% 28%

26% 24%

20%

44%

Name-and-address cleansing

Master data management

NO PLANS

18% 31%

55%

19%

54% 39%

22% 38%

Figure 11. Based on 318–337 respondents per answer.

This final section of the report discusses the available technology and tool types for CDI, following the order shown in Figure 11. Furthermore, all these tool types are available as products from software vendors, and so representative vendors and products are mentioned in the following discussion.4

Database Management Systems The DBMS is the most common enabling technology for CDI.

Database management systems (DBMSs) top the list of tools and technologies used for CDI today (82% in Figure 11). This makes sense, because databases are everywhere, from the applications in which most customer data originates to the analytic or operational applications that are fed by standardized customer data. Furthermore, many of the users interviewed for this report have taken a physical approach to CDI, in that they relocate customer data into some sort of mid-tier customer database. In a lot of ways, this is a data warehousing mentality applied to CDI. The resulting database varies a lot. With analytic CDI, it may be an enterprise data warehouse or customer data mart. With operational CDI, the mid-tier database is usually a custom operational data store (ODS). However, there’s a slow trend in CDI, away from persistent data in a mid-tier database and toward a more virtual or federated approach. Also, as MDM gives more methodology to CDI, relocating great volumes of customer data will hopefully give way to a more metadata- and master-data-driven approach. These trends explain why the future-oriented user priorities seen in Figure 7 ranked DBMSs lower than the current usage seen in Figure 11 did. Most application databases and mid-tier customer databases are running on the market-leading relational DBMSs from IBM, Microsoft, and Oracle.

Data Integration or ETL Data integration tools are very common in CDI solutions.

TDWI rese a rch

In second place by current usage (73%) are tools for data integration (DI) and extract, transform, and load (ETL). Tools for data integration are an obvious requirement, because the current practice of CDI moves and transforms a lot of data, most clearly seen in analytic CDI. ETL is the preferred type of data integration for data warehousing and, therefore, for analytic CDI, too. Put DBMSs and DI together, and you have the bulk of CDI’s enabling technologies. 4 The vendors and products mentioned here are representative, and the list is not intended to be comprehensive.

Vendor Tools and Platforms

Representative data integration tools (largely based on ETL) include BusinessObjects Data Integrator, Informatica PowerCenter, and Syncsort DM Express. But data integration can also take a federated approach, as seen in BusinessObjects Data Federator, Composite Information Server, and Denodo Virtual DataPort.5

BI Reporting and Analysis Tools Strictly speaking, BI reporting and analysis tools (66%) are not part of a CDI solution. Through these tools, BI end users consume the data prepared and delivered via CDI. Even so, survey respondents are mindful of the connection. On a related topic, note that CDI must deliver data in models that BI reporting and analysis tools demand. Most reporting tools just need relational data structures like tables and keys. But OLAP-based analysis tools may also need multidimensional structures. And analysis tools based on statistical or artificial intelligence methods (as are most data mining and predictive analytics tools) often require a very specific record structure.

CDI must deliver data in models reporting and analysis tools require.

Representative BI platforms are available from vendors like Actuate, Business Objects (SAP), Cognos (IBM), Hyperion (Oracle), Information Builders, Microsoft, MicroStrategy, and SAS.6

Data Warehouses and Marts With analytic CDI, data warehouses and marts (66%) are the expected targets. And CDI should transform data to models conducive to reporting and analysis. Data warehouses and data marts are typically running on the market-leading relational DBMSs mentioned above. But they may also run on DBMS platforms built specifically for data warehousing, like HP Neoview, Sybase IQ, and Teradata Warehouse. TDWI has noticed that most deployments of data warehouse appliances from DATAllegro and Netezza are used for multi-terabyte customer data marts. Note that some vendors supply packaged data models for data warehouses and marts used with CDI, such as BusinessObjects Rapid Mart.

Data Quality Tools As we saw earlier in this report, improving the quality of data is a common goal of CDI. Data quality tools support a variety of quality operations for name-and-address cleansing, match-and-merge, deduplication, verification, enhancement, standardization, and so on. They also include related capabilities for data discovery, data profiling, and data monitoring.

Over 40% of users plan to add more DQ tools to their CDI solution.

Let’s recall that DQ tools topped users’ priorities earlier in this report, yet they’re not in use today at the same level. For example, data cleansing and profiling (44%) and name-and-address cleansing (41%) ranked in the middle of the list by usage now. However, sizable percentages of survey respondents anticipate adding more tools of these types in the future, and responses for “no plans” are low. So, it looks like users will be bringing their DQ tool usage up to their priority levels in the near future. Many software vendors offer suites of DQ tools, like Acxiom CDI-X, BusinessObjects Data Quality, DataFlux dfPower Studio, Experian QAS, Informatica Data Quality, Informatica Identity Resolution (formerly Identity Systems), Silver Creek Systems DataLens System, and Trillium Software System.7

5 For a complete survey of data integration vendors and tools, see the TDWI Technology Market Report Data Integration Tools (Q4 2007), available to TDWI Members at www.tdwi.org/research. 6 For a comprehensive survey of dashboards (including vendors and products), see the TDWI Best Practices Report Deploying Dashboards and Scorecards, available online at www.tdwi.org/research/reportseries. 7 For a complete survey of data quality vendors and tools, see the TDWI Technology Market Report Enterprise Data Quality Tools (Q2 2006), available to TDWI Members at www.tdwi.org/research.

w w w.tdwi.org

customer data integration

Third-Party Customer Data Eschew myopia by integrating third-party customer data.

A common goal of CDI is to increase the effectiveness of data-driven sales and marketing activities. Integrating customer data from in-house sources is a step in the right direction toward the highly desirable 360-degree view of each customer. Yet, internal data tells you only what the customer does with your organization. To find out what the customer does elsewhere, you need to acquire thirdparty customer data from an external provider. For example, Acxiom’s InfoBase-X Enhancement provides customer data that can be appended to the customer records of in-house files and databases. The appended data gives customer-facing users greater insight, by providing data about customers’ demographics, socioeconomic brackets, and lifestyle behaviors. According to TDWI’s CDI survey, 41% of respondents already avail themselves of third-party customer data, while 28% plan to do so in the future.

Master Data Management Both master data and metadata are insufficiently applied to CDI solutions today.

If we recall that many users interviewed lamented that their CDI solutions lack MDM, it’s a relief to see that 26% of respondents to the CDI survey report including MDM functions today, which is much higher than TDWI had anticipated. And a whopping 55% plan to add MDM in the future. Although MDM is not used as commonly in CDI solutions as it should be today, users anticipate rectifying this situation soon. This is good, because MDM can give CDI one of its most coveted goals—consensus-driven definitions of the customer, applied consistently across multiple IT systems and departments. In terms of tools, most MDM solutions are custom-built in-house. TDWI’s MDM survey (run in late 2006) shows that most MDM solutions are homegrown from scratch (41%), though others can be developed on a vendor-built tool (25%) or both (29%). In fact, most MDM solutions are like CDI solutions, in that they are mostly constructed with a combination of database and DI tools.

Metadata Management Similar to master data management, metadata management is not used in CDI as much as it should be today (24%), although a lot of users anticipated using it more in the future (54%). Metadata management functions and repositories are embedded in all the aforementioned data integration and data quality tools. Exeros offers tools that discover cross-system business rules and metadata. Independent metadata tools and repositories include ASG Rochade and Computer Associates AllFusion Repository, which are designed for cross-application enterprise use.

Vendor Tools for CDI or MDM Vendor tools for CDI and MDM are numerous, but rarely used today.

TDWI rese a rch

As we saw in Figure 8, a combined 65% of CDI solutions are custom-built by IT or consultants. This helps explain why vendor tools for CDI or MDM (23%) ranked dead last by current usage in Figure 11. To make it worse, a high percentage of respondents (38%) say they’ve no plans for using CDI or MDM tools.

Vendor Tools and Platforms

Although usage is low today, there are many products available from software vendors. Tools focused on MDM infrastructure (upon which you or the vendor might build a CDI solution) include Exeros Discovery, Initiate Master Data Service Platform, Purisma Data Hub, SAP NetWeaver MDM, Siperian MDM Hub, and Teradata MDM. Tools focused on CDI include Acxiom CDI, DataFlux MDM Solution for Customer Data, IBM InfoSphere Master Data Management Server (formerly IBM WebSphere Customer Center), Oracle Customer Data Hub, and Siebel Universal Customer Master.8 USER STORY Winner of the 2008 TDWI Best Practices Award in Customer Intelligence Cisco Systems, Inc. designs, manufactures, and sells Internet protocol (IP) based networking and other products related to the communications information technology industry worldwide. Cisco won the TDWI 2008 Best Practices Award in Customer Intelligence for an internal IT initiative that achieved the four requirements of a TDWI award: business impact, maturity, relevance, and innovation. To read the stories of other award-winning users, visit www.tdwi.org/bpawards. Business Impact

Cisco’s new Customer Intelligence Center (CIC) is a framework and a platform based around an enterprise data warehouse (EDW) that focuses on customer data integration (CDI). The EDW integrates data from many sales, marketing, and financial applications, and serves up the data to support multiple analytic and operational applications. Through powerful application logic coupled with data models in the EDW (which integrates data across multiple domains), Cisco is able to correlate revenue bookings with sales and marketing activities. Due to this correlation, Cisco management knows that more than $500 million in bookings have been assisted by the CIC program, fully justifying the spend on this initiative. Maturity

Following deployment in June 2007, Cisco’s CIC had 250 users within a month, and in one year it has 1,700 active users logging into the application many times a week. User adoption is one of the most important metrics for gauging the success of an application, so a detailed deployment plan is in place to continue this success and double the adoption and usage in the next five months. Relevance

It’s been proposed that a 360-degree view of customers can increase sales through up-sell and cross-sell activities. In a lot of ways Cisco’s CIC proves this to be true, with the application using customer data and their interactions with Cisco to the fullest extent. Innovation

Cisco applied well-known data warehousing platforms, data integration tools, and reporting/analysis tools to their customer intelligence solution. The innovation is not so much in the selection of technologies, but rather in the diligent focus on tracking revenue in multiple dimensions, so that data can both enable sales and track sales accurately. This helps Cisco lift revenue, but also provides an understanding of customer needs that will lead to greater efficiency and effectiveness in sales and marketing.

8 For a survey of MDM vendors and tools, see the TDWI Technology Market Report Segmenting Master Data Management Solutions (Q4 2006),

online at www.tdwi.org/research. This report also sorts out the overlapping relationships among solutions for MDM, CDI, and product information management (PIM). For further distinctions of MDM and CDI, see Dyché and Levy, Ibid, Chapter 2.

w w w.tdwi.org

customer data integration

Recommendations

Once customer data is an enterprise asset, most goals fall into place.

Make customer data an enterprise asset. If you do this first, other goals fall in line more naturally. Once customer data is an enterprise asset, sharing it across organizational borders is done with little or no negotiating. Cross-system improvements in data, metadata, and master data get done faster and more accurately. And the 360-view of the customer comes into focus faster. Anticipate organizational dynamics. CDI’s severest barriers are ownership and turf issues. Data governance is good way to express an executive mandate, get line-of-business managers on board, and establish procedures for improving, integrating, and sharing customer data. Know the requirements of analytic CDI and operational CDI. They’re different, so you’ll need to decide whether to satisfy them with one or two solutions.

CDI has ROI and benefits, so just do it.

Promote CDI by its benefits. The three hottest areas for CDI benefits are BI, sales and marketing activities, and customer relations. Declare that CDI has ROI. It’s hard to see—because it’s indirect—but it’s there. Consolidate redundant CRM and CDI solutions. Put your customer data eggs in fewer baskets, so you have a more complete view of each customer and less synchronization of customer data across systems. If you have more than five CDI solutions, consider tightening controls on customer data. If you have more than 10, seek professional help.

Many firms must start by consolidating and upgrading existing CDI.

Admit that some of your CDI solutions are legacies. They need updating or replacement so they satisfy recent requirements for MDM, SOA, and bidirectional data distribution. Extend the scope of CDI. TDWI survey data shows that half of CDI solutions reach their entire enterprises. Make the other half do the same. Don’t initiate a new CDI solution without legal and compliance guidance. Customer data is sensitive, because it’s about human beings who are citizens. Comply with external regulations (especially federal and state laws) and internal policies (especially data privacy and security rules).

Data quality is the most critical technology piece.

Assume that data quality is critical. Users surveyed pegged DQ as their top priority, but also admitted that DQ is currently a small part of their CDI solutions. Increase DQ usage accordingly. Assess the quality of customer data. Customers obsessively change their addresses, phone numbers, jobs, economic brackets, preferences, and even their names. Due to the changeable nature of the customer, customer data suffers data quality problems systemically. Diligently monitor the quality of data and act appropriately to stay ahead of its inevitable degradation. Eschew myopia. Acquire third-party consumer data and integrate it into your CDI solution. Otherwise, you’ll see only what your customer does with you, whereas visibility into their financial brackets, demographics, consumer behaviors, and so on can suggest ways to retain and grow each customer. And external data is critical for accurate customer-base segmentations.

Don’t scrimp on data integration, because it’s CDI’s backbone.

TDWI rese a rch

Recognize that data integration (DI) is the leading enabler of CDI. After all, CDI is a specialized form of DI, and DI makes it happen more so than tools for databases, DQ, and MDM. Accordingly, beef up DI infrastructure and branch out beyond ETL to federated and real-time DI.