Regarding the Humpty Dumpty Data Warehouse Dilemma

Wayne Eckerson, on his Wayne’s World Blog for TDWI, revisits the dilemma of the “Humpty Dumpty Warehouse”:

Most organizations are like Humpty Dumpty teetering and tottering on top of a big wall. With the slightest gust of wind, Humpty crashes and breaks into dozens of pieces. And DW teams are “all the king’s horses and all the king’s men” who are charged with putting Humpty Dumpty back together again.

Whether we’re talking about “Humpty Dumpty” in terms of the enterprise as a whole, or the data within a given warehouse, agreed — DW teams are doing the best they can with what they have. But often so are the CEOs, who are facing battles in the boardroom, battles between the shareholders, the company’s bankers, the boards of directors, the various C-Levels within the organizations and some of their powerful subordinates. Never mind vacuums created when key executives leave, or when mergers and acquisitions, divestitures, etc. change the nature of the business.

Unfortunately, most current Data Warehouses are built in such a way that this Humpty Dumpty dilemma will repeat itself over and over again. The real dirty little secret is that the same tricks used to make DWs efficient for reporting purposes (aggregation, indexing, and the subsequent discarding of underlying details) are the ones that make them difficult — and expensive — to change and update.

So what’s the answer? First, we must realize that there is no such thing as a “single version of the truth” but merely a convenient and workable one. Next, we must break out of the “Humpty Dumpty” dilemma and its tragic ending and find a better story — a better model, like the “Phoenix” that can “rise from the ashes” overnight to meet all the new KPI’s to support the business needs.

The good news is that technological developments in Nearline 2.0, RDBMS federation capabilities and high-performance ETL tools offer a way for companies to transition from “Humpty Dumpty” to the new “Phoenix”-like approach — without resorting to a “rip and replace” strategy.

My next post will explore these ideas further.