Building Corporate Memory Into a Next-Generation Data Warehouse
It is now possible to design and implement a corporate memory within the data warehouse using a number of mature, tested and well-understood products and methodologies that can be deployed relatively quickly and administered with minimal DBA overhead. These solutions can grow with relatively linear scalability in terms of both cost and performance, while providing powerful support for both power analysts and reporting users. The ingredients for a successful data warehouse implementation that makes use of the corporate memory concept involve hardware, software and architectural design components, as listed below:
Data “Dumping Grounds” and the Importance of Corporate Memory
Received wisdom about data warehousing instructs us not to create a “dumping ground” for our raw detail data. But why not? This principle is a legacy from the not-so-distant past when it was impractical to keep huge amounts of data around if it was not being actively used – so once aggregates had been built, the original details were simply discarded. Of course, this meant that the organization was then confined to working with a particular “version of the truth” that someone had imposed on the data; there was no way to revisit the original details should the need arise for a change of perspective.
Decision Support for Users Who Don’t Know What They Don’t Know
Since the beginning of the computer era, system designers have struggled to reconcile conflicting aims of performance vs. functionality and maintainability vs. adaptability. In the case of Business Intelligence, there has been no less of a need for tradeoffs in order to deliver workable systems. However, BI system design has also typically been constrained even further by four fundamental realities:
Redefining the Role of IT in Business Intelligence
If our businesses are going to survive, we need to stop designing Business Intelligence systems that tell us what we want to hear, and which work well in good times but behave incomprehensibly during periods of significant inconsistency. Instead, we need to build systems that empower our best analysts to help correct flaws in our activities and identify opportunities that we can exploit. We need to be in a position where existing paradigms can be challenged and replaced by new ones on an ongoing basis. However, just as new scientific theories need to fit with observed reality, these new business approaches must be well supported by the facts as recorded in a company’s information repository.
Business Intelligence: An Oxymoron?
An old joke has it that the term “military intelligence” is an oxymoron – and in light of the current global financial crisis, it is tempting to put “Business Intelligence” in this category as well. Our inability to predict or deal competently with major events, from wars in the Middle East to the meltdown of global financial systems, shows just how ineffective our Business Intelligence/Data Warehousing strategies or “fit-for-purpose” reporting systems can be in responding to events as they unfold in this complex world. We are now confounded by the facts: we cannot predict the future; the largest military powers cannot conquer and control much weaker opponents; economists cannot adequately monitor essential financial systems. Automated trading systems, whose rules we once thought we understood and controlled, seem to have taken on a life of their own.
Nearline 2.0 vs. the Archive
In his most recent SAND blog post, Richard introduced the notion of “Nearline 2.0” and discussed how this concept, and related best practices, can be of vital importance to businesses dealing with the “data tsunami” we’ve been experiencing in recent years.
In this post, I’d like to step back a moment and explore the ways in which the dynamics of Nearline 2.0 differ from traditional methods of data archiving in terms of their approach to keeping data warehouse size under control.
2008 Season’s Greetings
To all of our friends and associates,
As we approach the end of the year, I would like to express my appreciation for your interest in and support of SAND Technology during 2007.
Read more…
Moving Beyond the Data Warehouse Impasse - Part 3
Once you extend your thinking beyond the data warehouse and “free” the data to speak for itself, the potential applications of the data un-warehouse concept are virtually unlimited. Let me suggest three powerful possible applications that would offer substantial benefit:
Read more…
Moving Beyond the Data Warehouse Impasse - Part 2
How can we move beyond the roadblock I described in my previous post? Let’s consider a powerful and positive alternative vision of corporate data resources – let’s call it the “data un-warehouse” scenario. This can be thought of as a data management regime that is the inverse of the data warehouse: data is “deconstructed” down to its elemental structure, free of indexes, dimensional modeling and complex schemas, and retained in a manner that is cost-effective but still readily available for exploration using standard Business Intelligence tools. Such a concept would not be realized as a technology that would replace the data warehouse, but rather would guide development of a parallel infrastructure that would have the effect of keeping the data warehouse “honest” while offering an additional portfolio of powerful analytic processes.
Read more…
Moving Beyond the Data Warehouse Impasse - Part 1
Over the last decade or so, data warehouses have expanded dramatically — both in size and in importance to the organizations that deploy them. The data warehouse has proved to be an invaluable means for transforming data created by production applications into information that is technically, structurally and organizationally ready for use by business managers and domain analysts. Yet like every successful technological advance, the data warehouse has its inherent limitations – and these are becoming more and more troublesome as the combined pressures of data volumes and user demands increase.
Read more…