Traditional vs. Agile BI Executive Summary INTRODUCTION Since Business Intelligence emerged into mainstream awareness in the 1990's, the imperative of delivering a "single version of the truth" has been an extremely challenging vision to realize in most organizations. Legacy BI has centered on the assumption that more information yields better decisions, and other than support for highly routine decisions made in mature, stable environments, this model has largely resulted in failure. This is mainly caused by three factors: 1) the difficulty and time required to integrate all of an organization’s data before any analysis can be done has resulted in extremely challenging implementations; 2) the fact that legacy BI technology has been optimized not for how decisions are made, but rather for solving technical limitations (many of which have been removed or are rapidly being removed); and 3) traditional BI platforms are unable to adapt to system change, which is inevitable given that we operate in a competitive environment where everything is dynamic. Traditional BI technologies have focused on solving data storage, integration, processing, and presentation issues. With the goal of decision support left unachieved, a new model of BI has emerged called Agile BI, which is built on many opposing assumptions like looser data integration, the utilization of less, more targeted information to make decisions, and the reality of continuous system change. Agile BI changes the focus from data driven to decision driven. MORE IS BETTER Legacy BI is based on the assumption that by having access to every piece of information about every aspect of a business process, we can make better decisions. This so called “single version of the truth” has lead to the ideal of the “Enterprise Data Warehouse”, in which a complete, unified view of our entire enterprise can be found. And by knowing everything, in context of everything else, our decision making will be fool proof. This is a very attractive idea that unfortunately just doesn’t work in practice. Even if it is theoretically possible to construct a unified, complete “single version of the truth,” it is likely the competition will have outmaneuvered you long before you are able to act upon it. And in competitive environment, the “truth” changes as new markets open, as new competitors come onto the field, and as market dynamics change the game.
This paper explains the fundamental assumption of traditional BI platforms that was made when business intelligence first emerged into the mainstream in the 1990s— and why it is no longer valid. Given this false assumption, we put forth the implications for how traditional platforms operate, and how this compares to more agile, lightweight platforms.
Not only is it impractical for most organizations to build a universal view, but current research in the decision sciences also indicates that decision models that attempt to include all available information actually don’t perform well in the real world. Such models do a great job of “predicting” the data you already have, but fail to work in new situations. And if you want humans to participate in the decisions, understand them, and take action, more information inputs and complexity leads to poorer adoption as well as difficulty in judging when the model might be failing. These factors have been echoed in poor BI adoption rates and the spectacular failures of so-called “data driven” organizations in the recent financial industry crises. Agile BI focuses on the requirements of the decisions being made, rather than on corralling all available data. Data may be tightly integrated to support decisions, or it may be loosely joined without the need for conformed dimensional models. As the decision model changes, information that seemed critical may fade in importance and new data source requirements will emerge. To be Agile, BI must quickly integrate (and disintegrate) this information for the decision maker. THE “TRUTH” CHANGES CONSTANTLY The success of Agile Methodology in software development is largely due to the fact that it accepts constant change as the norm. Every principle of that methodology is centered around delivering value-producing functionality quickly, and in a way that anticipates significant change in direction. Much like the software industry, BI has historically been plagued by constant changes in requirements. Anecdotes abound of end-users viewing a report for the first time and immediately responding with new requirements. But despite this, legacy BI architecture has failed to achieve any form of agility.
Traditional BI technologies, in assuming the goal of a single version of the truth, have focused largely on overcoming performance issues associated with meeting that goal. They have done so through a heavy-weight process of transforming data, dimensional modeling, summarizing or “cubing” data, creating metadata layers, etc. This architecture builds in key aspects of the decision process into every layer of the process. The implication of this is that even relatively simple requirement changes can trigger significant rework through the entire architecture. For example, changes to source systems, ETL jobs, the dimensional model, and the metadata often take months to deliver. When decisions are well known (“routine decisions”) such an architecture can support them. The requirements for basic financial reporting, for example, don’t change often. But many times, we need to make decisions that are novel, such as what markets to expand into, what products to introduce, or how to respond to a new competitor. These types of decisions will often require new information, and will often be very iterative in nature. The way the decision is made changes as the decision maker gains more information. And once made, such decisions can change the landscape entirely. This is not a job for legacy BI. The monolithic approach of legacy BI has actually led to desktop analysis tools (king of which is the spreadsheet) to become the standard in such decisions. When Oracle decides to acquire another BI vendor, that decision will be made in Excel, not OBIEE. Why? Agility. Agility to change the decision model on the fly. This monolithic architecture was required when 16-bit computing and nascent relational database technology made performance the primary barrier to decision support. With the emergence of 64-bit computing, columnar databases, cloud computing, and extreme data volume technologies like Hadoop, legacy BI architecture needlessly sacrifices agility to solve last century’s performance barriers. If BI is to be Agile, it must adopt an architecture that assumes constant change in requirements at all levels and is focused on the decision being made.
legacy BI likewise consistently fails to respond to these changes. Weinberger studies the impact of the internet on society and the impact of the internet in driving information globally has been the key information technology success of the last century. The internet was built on assumptions completely antithetical to legacy BI, focusing on providing very focused information that could be loosely joined to any other information stored anywhere in the world. And the move into Web 2.0 has taken the web from linking pages into a world in which we now are able to mash-up rich applications. Organizations have started to move away from wholesale adoption of full ERP packages and back to a best-of-breed approach supported by standards-based integration architectures such as SOAP-based web services. Capabilities can be added, changed, or removed without the need to completely re-architect the entire system, thus providing agility to the business. This architecture achieves the same type of flexibility that has made spreadsheets proliferate, but provides a powerful framework to avoid isolated, redundant and conflicting information silos. Agile BI will follow this model, allowing domain and decisionspecific information applications to be joined together to form BI platforms. These information applications will no longer be stand-alone “enterprise BI systems,” but will often appear embedded in the context of the transactional or other applications already is use. And as requirements change, applications will be added, removed, or updated quickly because they don’t require an assessment of their impact on a universal “single version of the truth” data model. Agility is about driving these changes in the marketplace. Think Wal-Mart or Amazon, but think paragons of Business Intelligence who stretch their implementations beyond traditional views of BI and use their insights to redefine and dominate their industry. In order to obtain and maintain such competitive positions, such organizations cannot wait for legacy architectures to catch up to emerging requirements. When you are redefining the competitive rules, you need decision support systems that can keep up.
AGILITY: DRIVING CHANGE, NOT RESPONDING
A NEW LANDSCAPE FOR BI
David Weinberger, a senior researcher at Harvard’s Berkman Center, talks about a phenomenon he calls the “changing shape of knowledge.” The idea is very much at the root of why BI requirements change so much, and in essence is a reflection that as we learn more, we tend to change the way we view what we knew in the first place. Most organizations struggle to keep up with the changing shape of knowledge in the marketplace and
The environment in which any decision support system must operate has completely transformed since the early attempts in the 1970’s to create an EIS. While legacy BI architectures continue to hold many of the same assumptions about information and computing that were true in the early 1990s, we’re seeing virtualization and cloud computing, Web 2.0 technologies, emerging standards and Services Oriented
7900 Westpark Drive, Suite T-107 | McLean, VA 22102 | Toll Free: 1-888-LOGIXML | www.logixml.com | info@logixml.com
Architectures, Advanced Analytics and Visual Analysis and a variety of other innovations that have completely changed the landscape. The current imperative in BI is to abandon the assumptions that have lead to such rigid solutions and leverage modern approaches to decision support that provide greater agility for the business. BI teams must move beyond legacy BI architectures and include technologies that support a rapid, iterative development style. The ability to rapidly source information, connect it to other information in both a tightly and loosely integrated fashion, and quickly connect BI applications together will be critical in meeting rapidly changing requirements.
7900 Westpark Drive, Suite T-107 | McLean, VA 22102 | Toll Free: 1-888-LOGIXML | www.logixml.com | info@logixml.com