Arthur, you are right to suggest that the BI team needs to adapt to changes Phoenix-like rather than pick up the pieces every time the organization changes. I guess the Humpty Dumpty metaphor is not the best–albeit a lot of fun–unless the king’s men are using superglue to get Humpty back together again. Certainly, I’m a big advocate of adaptable DW and BI architectures. That’s a given I should have noted!

Rather than superglue — though that sounds like fun! — last time I mentioned several key breakthroughs in information technologies that have matured to the point where a viable, flexible, Phoenix-like EDW can be created without taking a “rip and replace” strategy that would discard what has already been accomplished within the organization.

Because it maximizes existing investments both in products and people, this represents a much more secure and cost-effective route than trading a well understood set of problems for a replacement technology that may well solve some problems, but will inevitably replace them with a variety of altogether new. These breakthroughs include the following:

• Enhanced data base federation capabilities from all the major RDBMS providers, as well as from many Business Intelligence tool vendors like Business Objects.
• Very high-performance, storage-efficient and massively scalable software-based Nearline 2.0 storage systems that can house the entirety of an organization’s structured detail data, federated with the primary RDBMS.
• Very high-performance Column-Based Analytic Technology (CBAT) systems to support analytics for power users
• Very inexpensive and powerful desktop computers with adequate storage
• Relatively inexpensive blade servers
• Very high performance, efficient, automated ETL tools that can be used by the organization to set up and control the flow of data over time (including Disaster Recovery support using the Nearline 2.0 storage architecture).

It is now possible to integrate all of these subsystems into a single EDW architecture, resulting in a Scalable Corporate Information Factory (SCIF), to adapt Bill Inmon’s terminology). With this model in place, key BI analysts no longer need to focus primarily on transforming data to achieve a single version of “the truth” — in reality, an unattainable goal — to enable adequate performance, and more on on helping users derive real business value from corporate information by maximizing accessibility to “the facts” for the users who can provide essential business insights.

In subsequent posts I will explore this architecture in more detail.

Most organizations are like Humpty Dumpty teetering and tottering on top of a big wall. With the slightest gust of wind, Humpty crashes and breaks into dozens of pieces. And DW teams are “all the king’s horses and all the king’s men” who are charged with putting Humpty Dumpty back together again.

Whether we’re talking about “Humpty Dumpty” in terms of the enterprise as a whole, or the data within a given warehouse, agreed — DW teams are doing the best they can with what they have. But often so are the CEOs, who are facing battles in the boardroom, battles between the shareholders, the company’s bankers, the boards of directors, the various C-Levels within the organizations and some of their powerful subordinates. Never mind vacuums created when key executives leave, or when mergers and acquisitions, divestitures, etc. change the nature of the business.

Unfortunately, most current Data Warehouses are built in such a way that this Humpty Dumpty dilemma will repeat itself over and over again. The real dirty little secret is that the same tricks used to make DWs efficient for reporting purposes (aggregation, indexing, and the subsequent discarding of underlying details) are the ones that make them difficult — and expensive — to change and update.

So what’s the answer? First, we must realize that there is no such thing as a “single version of the truth” but merely a convenient and workable one. Next, we must break out of the “Humpty Dumpty” dilemma and its tragic ending and find a better story — a better model, like the “Phoenix” that can “rise from the ashes” overnight to meet all the new KPI’s to support the business needs.

The good news is that technological developments in Nearline 2.0, RDBMS federation capabilities and high-performance ETL tools offer a way for companies to transition from “Humpty Dumpty” to the new “Phoenix”-like approach — without resorting to a “rip and replace” strategy.

My next post will explore these ideas further.

• Hardware:
  • Cheap but powerful SMP hardware
  • Reasonably priced, fast and efficient SAN devices
  • Reasonably priced, fast and effective networks and switches for linking multiple SMP boxes together.
• Architecture:
  • Nearline data storage (not archiving) to hold the detail data used to build and maintain aggregates and cubes. There is a very clear differentiation between the Nearline 2.0 and archiving approaches to data and performance management.
  • Database federation techniques enabling reduction of the amount of data in key RDBMS “hot spots” and minimization of batch window requirements
  • Well-defined uses for aggregation, indexing, and MOLAP cubes to support reporting.
• Software:
  • Column-based data management technologies for avanced analytics, offering:
    • The ability to change data models “on the fly” to meet emerging requirements
    • Support for very wide tables (tens of thousands of columns), enabling very large numbers of KPIs
    • The ability to add new data types “on the fly”
    • The ability to allow existing applications to continue to work as data models change over time
    • The ability to present all available data in a simple, easily usable format, eliminating the need for analysts to navigate complex data relationships and slowly changing dimension constructs.
  • Better exploitation of new hardware and architectural techniques by traditional RDBMS systems
  • The ability to support massively parallel operations, allowing users to iteratively run very complex queries to find patterns in the data without having their thought process interrupted, without impacting other power users and other reporting users, and without interfering with the ability to meet critical SLAs.

Creating a “corporate memory” of this sort brings a number of major benefits:

• If it turns out that any or all of the original data is required for a new project (involving new aggregations, “cubes” or KPIs, for example), it is substantially less expensive and much easier to access it.
• If it becomes necessary to assess the value of the corporate data, this can be done carefully over a period of time using data mining techniques, rather than in a rush to meet specific project deadlines.
• The original detail data will only need to be handled once, and then can be retained as a permanent record for use in auditing the data warehouse if this becomes necessary (to prove compliance, to aid in conflict resolution, or simply to rectify “human errors” that were made at some point in the process)
• It becomes possible to document in the metadata layer who accessed the information, when and for what reason – incidentally, this can also provide valuable insight into the abilities of those who are responsible for manipulating the raw data.

In my next post, I’ll cover the ingredients for building Corporate Memory into a Next-Generation Data Warehouse.

• Users don’t know what they don’t know – they will always have difficulty articulating what they need, until they actually start working with the data.
• We in IT tend to want to implement relatively restrictive systems with well-defined acceptance criteria, in order to get buy-in from users. This protects us from backlash – and we know that users with looming deadlines will inevitably settle for less than they really require.
• Our IT training and experience has taught us that any system able to accommodate all the information that might be needed would be beyond our means – and even if it were affordable, it would be unlikely to perform adequately. As a result, we have gone to great lengths to determine the minimum amount of information required, then extracted that information and transformed it into the records that we retain.

However, it must be admitted that we almost never accurately judge what is “the minimum amount of information required” – and this typically leads to a series of costly attempts later on to find the missing data and retrofit it into our carefully designed data models.

• Generally, corporate executives have given up on the possibility of understanding the complexities of the computer world, and so have relinquished control of computer systems to the IT department. This lack of overarching leadership often results in a conflict situation, with the IT department facing various frustrated end-user groups who “don’t know what they don’t know”.

If IT departments are to make any progress in implementing Decision Support systems that really work, things need to change. We urgently need to approach the task of designing data warehouses in a spirit of humility and mutual co-operation. We must also acknowledge that some aspects of the resulting system may miss the mark – so we will need to build into the system a means of detecting and correcting problems, and for accepting user feedback and then efficiently incorporating the necessary adaptations into the system.

For a data warehouse environment to be of real value, it needs to be able to operate on the basis that users don’t know what they don’t know, by supporting the twists and turns of “incremental thinking” whereby they only gradually come to an understanding of their requirements. If a system designer comes right out and asks users what they want to know, they will typically be met with uncertainty. At this point, individuals anxious to promote their own way of thinking may step in with the assurance that they know what is really needed, while business experts who need to anticipate the effects of any number of potential scenarios from hurricanes to “credit crunches” often find themselves unable to make their voices heard. Locking in to a rigid, inflexible design on the basis of such inadequate input practically guarantees that the system will fail to deliver value over the long term.

In a Business Intelligence environment, incremental thinking usually involves “feeling one’s way” towards a goal by asking successive questions. Frequently, the approach will change – in terms of query strategies and the data sets we want to analyze – as insight is gained into the problem. Analysts work much like a detective will operate at the scene of a crime, picking up and examining various clues but not being able to link them together until the necessary understanding is developed. There are many spectacular examples of the enormous business value that can be derived from being able to ask any question that may arise in an analyst’s train of thought, particularly in highly dynamic activities like fuel hedging and currency trading. Success in these areas requires that plenty of historical details be available for unfettered and recursive access – allowing for the types of question that often provide the biggest business benefits.

However, it is obviously not necessary or even feasible to scan all the details every time a question needs to be answered. For this reason, a combination of traditional reporting and analysis architectures with a Nearline 2.0 repository (used to store massive amounts of detail data) can be the ideal solution. This architecture takes advantage of breakthroughs in database federation techniques and cheap, scalable SMP processors to create a massively parallel architecture that can easily scale to meet future needs, with little or no chance of technological obsolescence. In future posts I will look more closely at how such an architecture can be designed and implemented to enable high-performance, flexible decision support for business “at the speed of thought”.

The IT department cannot continue to dictate how IT-held data will be used: users have to be empowered to make these decisions themselves within their area of primary expertise. The focus of IT should instead be on making data available to end users, ensuring that it is not corrupted and that it can be audited when required, and on infrastructural issues like security, disaster recovery and so on. The job of the IT department should thus go far beyond merely distributing preconfigured reports, to encompass the more essential task of empowering users to get what they really need, when and how they need it, and to make sure that adequate resources are in place to get the job done. In short, IT departments need to work in a similar way to utility companies, providing users with infrastructure options but not with final solutions – and as with a power company, for example, fair and appropriate charge-out policies and procedures need to be put in place, so that IT can be transformed from a cost centre into a profit centre.

So, the primary role of IT should be to provide unfettered access to data, to make sure that the data will not be lost for any reason (even natural disasters), and to meet various levels of demand. While security has traditionally been another key issue for IT, even this area might be better handled by a specialized department, with IT providing assistance as needed. In any event, IT should not be able to prevent anyone with the appropriate security levels (which in some cases may exceed those of the head of IT) from getting to the data they need. Often, invocations of security or the impact on performance for “other system users” are simply a ruse to cover-up for the fact that the data is not readily available or that the system is not designed to cope with the demand.

We need to understand that a functional Business Intelligence (BI) system is governed by the same rules that govern any other large architectural undertaking like an office building or an elevated highway. It must meet the diversified needs of the community it serves and those of the people whose lives it affects. It also has to fit within the constraints already in place in the environment in which it is to be established – since very few of us get to work with a “clean slate”, and the legal and administrative hurdles can often be formidable.

Successful implementation of BI systems thus needs to be considered as part art and part science. Pure science has not been able to solve all society’s problems, and those who try to adopt a purely “scientific” approach to BI will not be successful. Fundamentally, the field of technology is not just about pursuing technical innovation, but about helping human beings become more effective and productive in pursuing their goals. In my following posts, I will be looking at how IT departments can go about designing systems that allow for just this sort of flexible, on-demand provision of information services to analysts, with the ultimate aim of enabling truly effective decision support for maximum repsonsiveness in today’s unpredictable business environment.

Arthur Ritchie

In this series of blog posts, I will be discussing how an effective information infrastructure for decision support can be implemented without resorting to a “rip and replace” strategy that would involve completely scrapping the existing data warehouse. As the foundation of an optimal system for enterprise data management and decision support, I will be proposing the concept of Data Warehouse as a tiered architecture that combines three database models/technologies to support both standard reporting and “power analytics” as well as highly accessible storage of massive amounts of granular data to be used by the reporting and analytics engines.

Arthur Ritchie

For a data framework solution like Nearline 2.0, XAM provides the ability to store large amounts of structured data (for example, call detail records, application logs, web logs, syslogs, low-level manufacturing data, data exported from an enterprise data warehouse such as SAP BW, and so on) securely and efficiently to ensure compliance with data governance requirements. SAND/DNA Access can use XAM storage providers like EMC Centera as WORM (Write Once Read Many) devices, ensuring that stored data can not be modified. Furthermore, the Centera XAM compliance solution offers robust integrated business continuity and disaster recovery protection for structured data in SAND/DNA Access. In turn, SAND/DNA Access maintains massive volumes of structured data as content for the XAM storage provider, enabling a complete enterprise solution for data governance and compliance. SAND/DNA Access also brings high-performance query capability for structured data in the XAM storage provider, making it possible to conduct true electronic discovery (e-discovery) on this data. All these considerations contributed to SAND’s decision to implement support for the new XAM API.

We began development of XAM API support at the beginning of October 2008, and within a month were ready to execute our benchmark tests at the EMC Hopkinton Lab. These tests, which were fully supported by the EMC Centera team, allowed us to validate our XAM API implementation and to evaluate the performance of the data migration process (moving data from SAND/DNA Access to the Centera server) as well as query speeds. In both cases, observed performance was very impressive: using limited processing power (4 blade servers), we were able to achieve data migration speeds higher than 400 MB per second, and query speeds of close to 11 million records scanned per second! This shows that the new XAM functionality of SAND/DNA Access is more than up to the task of enabling existing Centera or XAM storage provider customers to extend their infrastructure to incorporate data compliance and governance regimes for massive amounts of structured data.

Let’s start at the beginning of the process. In a traditional data warehouse implementation, data is extracted from operational systems, then transformed and loaded into the data warehouse. Figure 1 shows this process, usually called ETL (Extract, Transform and Load) or sometimes ELT (Extract, Load and Transform), in the context of the Corporate Information Factory.

In previous posts, I presented Nearline 2.0 as a “database extension” that can reduce TCO, guarantee satisfaction of SLAs, and improve the performance of the online database that underpins the data warehouse solution. This configuration is based on the traditional concept of Information Lifecycle Management (ILM), normally implemented as a tactical effort to protect the enterprise against problems associated with the “data tsunami”. Figure 2 illustrates the incorporation of Nearline 2.0 into the data management framework as a database extension used to store (and maintain access to) older or less frequently used detail data.

While this ILM concept is a sound one, it has important limitations because of its reliance on migrating data from the online database to the nearline repository, with all the technical and human resource requirements this entails. Is it really necessary to do it this way? In my last post, I introduced the notion of Just-in-Case (JIC) data, which involves maintaining access to the detail data that underlies data warehouse summary tables. If the JIC data (which is static by definition) is “nearlined” at creation time — or as as soon as possible after it is created, — we can avoid the costs and system impact of moving it into the data warehouse. Why should we migrate JIC data to the online data warehouse, only to face the additional costs and hassles of archiving it later on? This will be a vital consideration when strategic decisions are being made about the enterprise data management framework.

It clearly makes much more sense for the data to be stored directly in the Nearline 2.0 repository. At SAND, we call this type of architecture, which is only possible because of the availability of Nearline 2.0 technology, the Corporate Information Memory (CIM). This is illustrated in figure 3.

The key benefit of this architecture for any enterprise is that it is based on an evolution of the current architecture, and so protects existing investments in people skills, software and equipment. This fits perfectly with Gartner’s observations in a recent report on “Key Issues for Delivering a Data Warehouse Project, 2008”:

“The big theme for data warehousing in 2008 is the increased demand for more data, in more places and doing so with an evolutionary approach. Data warehouses are mission-critical and integrated into operations. Failure to support business process change and inflexibility will not be tolerated. The problem is how to take the years of effort and funds invested previously and leverage them into a modern data warehouse because “rip and replace” strategies are not acceptable.”

So, it is not difficult to see how Nearline 2.0 can act as a pivotal element in an enterprise data management framework, and how it presents one of the best available solutions to the critical issues that have arisen in this area. In my next post, I will discuss the benefits of this type of architecture in more detail.

Richard Grondin

The Physical Table Partitioning Model

The first of these new data modeling options is based on physical table partitioning. The largest tables in a data warehouse or data mart can be physically divided between an online component and a Nearline counterpart. This allows the existing data model to be maintained, while introducing a “right-sizing” concept where only the regularly accessed data is kept online, and all data that doesn’t require such an expensive and/or hard to manage environment is put into the Nearline 2.0 solution. A typical rule of thumb for defining partition boundaries is based on the 90-day aging principle, so that any static data older than 90 days is migrated from the online warehouse to the Nearline 2.0 repository.

Now, many forms of enterprise data, such as CDR, POS, Web, Proxy or Log data, are static by definition, and are furthermore usually the main sources of data warehouse growth. This is very good news, because it means that as soon as the data is captured, it can be moved to the Nearline 2.0 repository (in fact, it is conceivable that this kind of data could be fed directly to Nearline 2.0 from the source system – but that is a topic for another post). Because of the large volumes involved, this kind of detail data has usually been aggregated at one or more levels in the enterprise data warehouse. Users generally query the summary table in order to identify trends, only drilling down into the details for a specific range of records when specific needs or opportunities are identified. This data access technique is well known, and has been in use for quite some time.

The Online Summary Table Model

This leads me to the second novel design option offered by Nearline 2.0: the ability to store all static detail data in the Nearline repository, and then use this as the basis for building online summary tables, with the ability to quickly drill to detail in the Nearline 2.0 repository when required. More specifically, the Nearline 2.0 solution can be used to feed the online system’s summary tables directly. The advantage of this implementation is that it substantially reduces the size of the online database, optimizes its performance, and permits trend analysis on even very long periods. This is particularly useful when looking for emerging trends (positive or negative) related to specific products or offerings, because it gives managers the chance to analyze and respond to issues and opportunities within a realistic time frame. A recent press release from Experian provides an excellent example of how business value can be produced by expert analysts using long-term historical data and the right set of tools.

Some organizations are already building this type of data hierarchy, using Data Marts or analytic cubes fed by the main Data Warehouse. I call this kind of architecture “data pipelining”. Nearline 2.0 can play an important role in such an implementation, since its repository can be shared between all the analytic platforms. This not only reduces data duplication, management/operational overhead, and requirements for additional hardware and software, it also relieves pressure on batch windows and lowers the risk of data being out of synch. Furthermore, this implementation can assist organizations with data governance and Master Data Management while improving overall data quality.

The Just-In-Case Data Model

Another important data modeling option offered by Nearline 2.0 relates to what we can call “just-in-case” data. In many cases, certain kinds of data will also be maintained outside the warehouse just in case an analyst requires ad hoc access to it for a specific study. Sometimes, for convenience, this “exceptional” data is stored in the data warehouse. However, keeping this data in an expensive storage and software environment, or even storing it on tape or inexpensive disks as independent files, can create a data management nightmare. At the same time, studies demonstrate that a very large portion of the costs associated with ad hoc analysis are concentrated in the data preparation phase. As part of this phase, the analyst needs to “shop” for the just-in-case data to be analyzed, meaning that he or she needs to find , “slice”, clean, transform and use it to build a temporary analytic platform, sometimes known as an Exploration Warehouse or “Exploration Mart.

Nearline 2.0 can play a very important role in such a scenario. Just-in-case data can be stored in the Nearline repository, and analysts can then query it directly using standard SQL-based front-end tools to extract, slice and prepare the data for analytic use. Since much less time is spent on data preparation, far more time is available for data analysis — and there is no impact on the performance of the main reporting system. This acceleration of the data preparation phase results from the availability of a central catalog describing all the available data. The Nearline repository can be used to directly feed the expert’s preferred analytic platform, generally resulting in a substantial improvement in analyst productivity. Analysts can focus on executing their analyses, and on bringing more value to the enterprise, rather than on struggling to get access to clean and reliable data.

At SAND, we have developed a new analytical offering based on this approach: “Database on Demand” or DBOD. Using DBOD, an analyst can specify their data requirements and have a 100% indexed RDBMS built directly from these specifications, ready for querying by front-end tools such as SAS, SPSS, Business Objects, MicroStrategy and so on. I’ll touch on that more in a future post.

Richard Grondin