Evolving the “Humpty Dumpty Warehouse” Into a “Phoenix”

In my last blog post, I responded to Wayne Eckerson Wayne’s World Blog for TDWI, which revisited the dilemma of the “Humpty Dumpty Warehouse”. I suggested that the “Phoenix” might be a better model for modern enterprise data warehouses. Wayne continued the discussion in comments:

Arthur, you are right to suggest that the BI team needs to adapt to changes Phoenix-like rather than pick up the pieces every time the organization changes. I guess the Humpty Dumpty metaphor is not the best–albeit a lot of fun–unless the king’s men are using superglue to get Humpty back together again. Certainly, I’m a big advocate of adaptable DW and BI architectures. That’s a given I should have noted!

Rather than superglue — though that sounds like fun! — last time I mentioned several key breakthroughs in information technologies that have matured to the point where a viable, flexible, Phoenix-like EDW can be created without taking a “rip and replace” strategy that would discard what has already been accomplished within the organization.

Because it maximizes existing investments both in products and people, this represents a much more secure and cost-effective route than trading a well understood set of problems for a replacement technology that may well solve some problems, but will inevitably replace them with a variety of altogether new. These breakthroughs include the following:

• Enhanced data base federation capabilities from all the major RDBMS providers, as well as from many Business Intelligence tool vendors like Business Objects.
• Very high-performance, storage-efficient and massively scalable software-based Nearline 2.0 storage systems that can house the entirety of an organization’s structured detail data, federated with the primary RDBMS.
• Very high-performance Column-Based Analytic Technology (CBAT) systems to support analytics for power users
• Very inexpensive and powerful desktop computers with adequate storage
• Relatively inexpensive blade servers
• Very high performance, efficient, automated ETL tools that can be used by the organization to set up and control the flow of data over time (including Disaster Recovery support using the Nearline 2.0 storage architecture).

It is now possible to integrate all of these subsystems into a single EDW architecture, resulting in a Scalable Corporate Information Factory (SCIF), to adapt Bill Inmon’s terminology). With this model in place, key BI analysts no longer need to focus primarily on transforming data to achieve a single version of “the truth” — in reality, an unattainable goal — to enable adequate performance, and more on on helping users derive real business value from corporate information by maximizing accessibility to “the facts” for the users who can provide essential business insights.

In subsequent posts I will explore this architecture in more detail.