Building Corporate Memory Into a Next-Generation Data Warehouse
It is now possible to design and implement a corporate memory within the data warehouse using a number of mature, tested and well-understood products and methodologies that can be deployed relatively quickly and administered with minimal DBA overhead. These solutions can grow with relatively linear scalability in terms of both cost and performance, while providing powerful support for both power analysts and reporting users. The ingredients for a successful data warehouse implementation that makes use of the corporate memory concept involve hardware, software and architectural design components, as listed below:
Data “Dumping Grounds” and the Importance of Corporate Memory
Received wisdom about data warehousing instructs us not to create a “dumping ground” for our raw detail data. But why not? This principle is a legacy from the not-so-distant past when it was impractical to keep huge amounts of data around if it was not being actively used – so once aggregates had been built, the original details were simply discarded. Of course, this meant that the organization was then confined to working with a particular “version of the truth” that someone had imposed on the data; there was no way to revisit the original details should the need arise for a change of perspective.
Decision Support for Users Who Don’t Know What They Don’t Know
Since the beginning of the computer era, system designers have struggled to reconcile conflicting aims of performance vs. functionality and maintainability vs. adaptability. In the case of Business Intelligence, there has been no less of a need for tradeoffs in order to deliver workable systems. However, BI system design has also typically been constrained even further by four fundamental realities:
Redefining the Role of IT in Business Intelligence
If our businesses are going to survive, we need to stop designing Business Intelligence systems that tell us what we want to hear, and which work well in good times but behave incomprehensibly during periods of significant inconsistency. Instead, we need to build systems that empower our best analysts to help correct flaws in our activities and identify opportunities that we can exploit. We need to be in a position where existing paradigms can be challenged and replaced by new ones on an ongoing basis. However, just as new scientific theories need to fit with observed reality, these new business approaches must be well supported by the facts as recorded in a company’s information repository.
Business Intelligence: An Oxymoron?
An old joke has it that the term “military intelligence” is an oxymoron – and in light of the current global financial crisis, it is tempting to put “Business Intelligence” in this category as well. Our inability to predict or deal competently with major events, from wars in the Middle East to the meltdown of global financial systems, shows just how ineffective our Business Intelligence/Data Warehousing strategies or “fit-for-purpose” reporting systems can be in responding to events as they unfold in this complex world. We are now confounded by the facts: we cannot predict the future; the largest military powers cannot conquer and control much weaker opponents; economists cannot adequately monitor essential financial systems. Automated trading systems, whose rules we once thought we understood and controlled, seem to have taken on a life of their own.
Using XAM with Nearline 2.0 to Ensure Data Compliance
Recently, SAND has been conducting tests on our SAND/DNA Access product to benchmark its support for the XAM (eXtensible Access Method) API. These tests, executed at EMC’s lab in Hopkinton, Mass. using the latest version of EMC’s Centera solution, were a great success in a number of respects, and the SAND/DNA Access Nearline 2.0 software component is now the first commercial product to obtain XAM certification from EMC. In today’s blog post, I will describe the XAM interface, explain our motivation for implementing it, and provide some details about the benchmark tests.
Nearline 2.0 and the Data Management Framework
In my last post, I outlined some of the advanced data modeling options that have been made possible by the advent of Nearline 2.0. Today, I want to discuss how Nearline 2.0 can act as an essential component in a data management framework. The data management framework, which can be viewed as an extension of the data model concept to the level of enterprise data architecture, governs the processing and management of enterprise data throughout its “lifecycle”, from creation to disposal. It includes all operational components, and covers key issues such as data backup, disaster recovery, data retention, data access security and so on.
Nearline 2.0 and Advanced Data Modeling
In my last post, I discussed the “Quick Check” method for identifying the benefits that a Nearline 2.0 implementation can deliver in the areas of operational performance, SLAs and TCO. Certainly, it is easy to see how it would be preferable to manage a database that is 1 TB rather than 20 TB in size, particularly when it comes to critical tasks like backup and recovery, disaster recovery, off-site backups and historical analytics. Today, however, I want to focus on another benefit of Nearline 2.0 that is less obvious but still very important: data modeling flexibility.
Starting Nearline 2.0: The Quick Check Approach
In previous posts, I introduced the concept of “Best Practices” for Nearline 2.0. Today, I will get down to the details of how and where to start with a Nearline 2.0 solution, beginning with a Best Practices approach designed to quickly identify the benefits of such an implementation in a given environment. At SAND Technology, we offer this “Nearline 2.0 Quick Check” as part of our professional services portfolio.
Nearline 2.0 Best Practices
In previous posts, we introduced the concept of Nearline 2.0, showed how it represented a significant step forward from traditional archiving practices, and discussed how Nearline 2.0 could help your business. To recapitulate: the major advantage of Nearline 2.0 is its superior data access performance, which enables a more aggressive approach to migrating data out of the online repository to nearline (a process known as “data nearlining”) than is practical when using a traditional archiving product.
Read more…