SAND News » Arthur’s Blog

Evolving the “Humpty Dumpty Warehouse” Into a “Phoenix”

admin — Mon, 26 Oct 2009 15:43:51 +0000

In my last blog post, I responded to Wayne Eckerson Wayne’s World Blog for TDWI, which revisited the dilemma of the “Humpty Dumpty Warehouse”. I suggested that the “Phoenix” might be a better model for modern enterprise data warehouses. Wayne continued the discussion in comments:

Arthur, you are right to suggest that the BI team needs to adapt to changes Phoenix-like rather than pick up the pieces every time the organization changes. I guess the Humpty Dumpty metaphor is not the best–albeit a lot of fun–unless the king’s men are using superglue to get Humpty back together again. Certainly, I’m a big advocate of adaptable DW and BI architectures. That’s a given I should have noted!

Rather than superglue — though that sounds like fun! — last time I mentioned several key breakthroughs in information technologies that have matured to the point where a viable, flexible, Phoenix-like EDW can be created without taking a “rip and replace” strategy that would discard what has already been accomplished within the organization.

Because it maximizes existing investments both in products and people, this represents a much more secure and cost-effective route than trading a well understood set of problems for a replacement technology that may well solve some problems, but will inevitably replace them with a variety of altogether new. These breakthroughs include the following:

Enhanced data base federation capabilities from all the major RDBMS providers, as well as from many Business Intelligence tool vendors like Business Objects.
Very high-performance, storage-efficient and massively scalable software-based Nearline 2.0 storage systems that can house the entirety of an organization’s structured detail data, federated with the primary RDBMS.
Very high-performance Column-Based Analytic Technology (CBAT) systems to support analytics for power users
Very inexpensive and powerful desktop computers with adequate storage
Relatively inexpensive blade servers
Very high performance, efficient, automated ETL tools that can be used by the organization to set up and control the flow of data over time (including Disaster Recovery support using the Nearline 2.0 storage architecture).

It is now possible to integrate all of these subsystems into a single EDW architecture, resulting in a Scalable Corporate Information Factory (SCIF), to adapt Bill Inmon’s terminology). With this model in place, key BI analysts no longer need to focus primarily on transforming data to achieve a single version of “the truth” — in reality, an unattainable goal — to enable adequate performance, and more on on helping users derive real business value from corporate information by maximizing accessibility to “the facts” for the users who can provide essential business insights.

In subsequent posts I will explore this architecture in more detail.

Regarding the Humpty Dumpty Data Warehouse Dilemma

Arthur — Sat, 26 Sep 2009 14:25:58 +0000

Wayne Eckerson, on his Wayne’s World Blog for TDWI, revisits the dilemma of the “Humpty Dumpty Warehouse”:

Most organizations are like Humpty Dumpty teetering and tottering on top of a big wall. With the slightest gust of wind, Humpty crashes and breaks into dozens of pieces. And DW teams are “all the king’s horses and all the king’s men” who are charged with putting Humpty Dumpty back together again.

Whether we’re talking about “Humpty Dumpty” in terms of the enterprise as a whole, or the data within a given warehouse, agreed — DW teams are doing the best they can with what they have. But often so are the CEOs, who are facing battles in the boardroom, battles between the shareholders, the company’s bankers, the boards of directors, the various C-Levels within the organizations and some of their powerful subordinates. Never mind vacuums created when key executives leave, or when mergers and acquisitions, divestitures, etc. change the nature of the business.

Unfortunately, most current Data Warehouses are built in such a way that this Humpty Dumpty dilemma will repeat itself over and over again. The real dirty little secret is that the same tricks used to make DWs efficient for reporting purposes (aggregation, indexing, and the subsequent discarding of underlying details) are the ones that make them difficult — and expensive — to change and update.

So what’s the answer? First, we must realize that there is no such thing as a “single version of the truth” but merely a convenient and workable one. Next, we must break out of the “Humpty Dumpty” dilemma and its tragic ending and find a better story — a better model, like the “Phoenix” that can “rise from the ashes” overnight to meet all the new KPI’s to support the business needs.

The good news is that technological developments in Nearline 2.0, RDBMS federation capabilities and high-performance ETL tools offer a way for companies to transition from “Humpty Dumpty” to the new “Phoenix”-like approach — without resorting to a “rip and replace” strategy.

My next post will explore these ideas further.

Building Corporate Memory Into a Next-Generation Data Warehouse

Arthur — Fri, 22 May 2009 13:00:46 +0000

It is now possible to design and implement a corporate memory within the data warehouse using a number of mature, tested and well-understood products and methodologies that can be deployed relatively quickly and administered with minimal DBA overhead. These solutions can grow with relatively linear scalability in terms of both cost and performance, while providing powerful support for both power analysts and reporting users. The ingredients for a successful data warehouse implementation that makes use of the corporate memory concept involve hardware, software and architectural design components, as listed below:

Hardware:
- Cheap but powerful SMP hardware
- Reasonably priced, fast and efficient SAN devices
- Reasonably priced, fast and effective networks and switches for linking multiple SMP boxes together.
Architecture:
- Nearline data storage (not archiving) to hold the detail data used to build and maintain aggregates and cubes. There is a very clear differentiation between the Nearline 2.0 and archiving approaches to data and performance management.
- Database federation techniques enabling reduction of the amount of data in key RDBMS “hot spots” and minimization of batch window requirements
- Well-defined uses for aggregation, indexing, and MOLAP cubes to support reporting.
Software:
- Column-based data management technologies for avanced analytics, offering:
  - The ability to change data models “on the fly” to meet emerging requirements
  - Support for very wide tables (tens of thousands of columns), enabling very large numbers of KPIs
  - The ability to add new data types “on the fly”
  - The ability to allow existing applications to continue to work as data models change over time
  - The ability to present all available data in a simple, easily usable format, eliminating the need for analysts to navigate complex data relationships and slowly changing dimension constructs.
- Better exploitation of new hardware and architectural techniques by traditional RDBMS systems
- The ability to support massively parallel operations, allowing users to iteratively run very complex queries to find patterns in the data without having their thought process interrupted, without impacting other power users and other reporting users, and without interfering with the ability to meet critical SLAs.

Together, these components can work within a next-generation data warehouse design that supports standard reporting while also giving expert analysts the ability to access the raw “truth” of the original detail data. This makes it possible for the organization’s most creative thinkers to extend their thought processes “outside the box”, to challenge existing dogmas and provide alternative and hopefully more useful versions of “the truth”. In my subsequent posts, I will look more closely at the details of these various architectural components and elaborate on how each of them can work to contribute to a powerful, flexible, and efficient decision support infrastructure.

Data “Dumping Grounds” and the Importance of Corporate Memory

Arthur — Mon, 04 May 2009 15:34:30 +0000

Received wisdom about data warehousing instructs us not to create a “dumping ground” for our raw detail data. But why not? This principle is a legacy from the not-so-distant past when it was impractical to keep huge amounts of data around if it was not being actively used – so once aggregates had been built, the original details were simply discarded. Of course, this meant that the organization was then confined to working with a particular “version of the truth” that someone had imposed on the data; there was no way to revisit the original details should the need arise for a change of perspective.

Now, however, Nearline 2.0 technology allows us to store massive quantities of data in an easily acccessible format without undue administration requirements, such an approach is economically viable not only in terms of storage and retrieval, but also from a human resources perspective.

Creating a “corporate memory” of this sort brings a number of major benefits:

If it turns out that any or all of the original data is required for a new project (involving new aggregations, “cubes” or KPIs, for example), it is substantially less expensive and much easier to access it.
If it becomes necessary to assess the value of the corporate data, this can be done carefully over a period of time using data mining techniques, rather than in a rush to meet specific project deadlines.
The original detail data will only need to be handled once, and then can be retained as a permanent record for use in auditing the data warehouse if this becomes necessary (to prove compliance, to aid in conflict resolution, or simply to rectify “human errors” that were made at some point in the process)
It becomes possible to document in the metadata layer who accessed the information, when and for what reason – incidentally, this can also provide valuable insight into the abilities of those who are responsible for manipulating the raw data.

In my next post, I’ll cover the ingredients for building Corporate Memory into a Next-Generation Data Warehouse.

Decision Support for Users Who Don’t Know What They Don’t Know

Arthur — Fri, 24 Apr 2009 13:10:46 +0000

Since the beginning of the computer era, system designers have struggled to reconcile conflicting aims of performance vs. functionality and maintainability vs. adaptability. In the case of Business Intelligence, there has been no less of a need for tradeoffs in order to deliver workable systems. However, BI system design has also typically been constrained even further by four fundamental realities:

Users don’t know what they don’t know – they will always have difficulty articulating what they need, until they actually start working with the data.

We in IT tend to want to implement relatively restrictive systems with well-defined acceptance criteria, in order to get buy-in from users. This protects us from backlash – and we know that users with looming deadlines will inevitably settle for less than they really require.

Our IT training and experience has taught us that any system able to accommodate all the information that might be needed would be beyond our means – and even if it were affordable, it would be unlikely to perform adequately. As a result, we have gone to great lengths to determine the minimum amount of information required, then extracted that information and transformed it into the records that we retain.

However, it must be admitted that we almost never accurately judge what is “the minimum amount of information required” – and this typically leads to a series of costly attempts later on to find the missing data and retrofit it into our carefully designed data models.

Generally, corporate executives have given up on the possibility of understanding the complexities of the computer world, and so have relinquished control of computer systems to the IT department. This lack of overarching leadership often results in a conflict situation, with the IT department facing various frustrated end-user groups who “don’t know what they don’t know”.

If IT departments are to make any progress in implementing Decision Support systems that really work, things need to change. We urgently need to approach the task of designing data warehouses in a spirit of humility and mutual co-operation. We must also acknowledge that some aspects of the resulting system may miss the mark – so we will need to build into the system a means of detecting and correcting problems, and for accepting user feedback and then efficiently incorporating the necessary adaptations into the system.

For a data warehouse environment to be of real value, it needs to be able to operate on the basis that users don’t know what they don’t know, by supporting the twists and turns of “incremental thinking” whereby they only gradually come to an understanding of their requirements. If a system designer comes right out and asks users what they want to know, they will typically be met with uncertainty. At this point, individuals anxious to promote their own way of thinking may step in with the assurance that they know what is really needed, while business experts who need to anticipate the effects of any number of potential scenarios from hurricanes to “credit crunches” often find themselves unable to make their voices heard. Locking in to a rigid, inflexible design on the basis of such inadequate input practically guarantees that the system will fail to deliver value over the long term.

In a Business Intelligence environment, incremental thinking usually involves “feeling one’s way” towards a goal by asking successive questions. Frequently, the approach will change – in terms of query strategies and the data sets we want to analyze – as insight is gained into the problem. Analysts work much like a detective will operate at the scene of a crime, picking up and examining various clues but not being able to link them together until the necessary understanding is developed. There are many spectacular examples of the enormous business value that can be derived from being able to ask any question that may arise in an analyst’s train of thought, particularly in highly dynamic activities like fuel hedging and currency trading. Success in these areas requires that plenty of historical details be available for unfettered and recursive access – allowing for the types of question that often provide the biggest business benefits.

However, it is obviously not necessary or even feasible to scan all the details every time a question needs to be answered. For this reason, a combination of traditional reporting and analysis architectures with a Nearline 2.0 repository (used to store massive amounts of detail data) can be the ideal solution. This architecture takes advantage of breakthroughs in database federation techniques and cheap, scalable SMP processors to create a massively parallel architecture that can easily scale to meet future needs, with little or no chance of technological obsolescence. In future posts I will look more closely at how such an architecture can be designed and implemented to enable high-performance, flexible decision support for business “at the speed of thought”.

Redefining the Role of IT in Business Intelligence

Arthur — Mon, 06 Apr 2009 16:00:42 +0000

If our businesses are going to survive, we need to stop designing Business Intelligence systems that tell us what we want to hear, and which work well in good times but behave incomprehensibly during periods of significant inconsistency. Instead, we need to build systems that empower our best analysts to help correct flaws in our activities and identify opportunities that we can exploit. We need to be in a position where existing paradigms can be challenged and replaced by new ones on an ongoing basis. However, just as new scientific theories need to fit with observed reality, these new business approaches must be well supported by the facts as recorded in a company’s information repository.

The recent crisis in financial systems, resulting in major corporations going cap-in-hand to governments for handouts, dramatically illustrates why we need to move beyond reliance on reporting systems to adoption of a challenge-based system. With a challenge-based system, key analysts and “out-of-the-box” thinkers with sufficient analytic capability and insights are given unconstrained access to the data warehouse so that they can identify problems and recommend corrective action based on the facts. In this way, the people currently controlling the “spin” on corporate information can be educated (or bypassed if necessary), and executives can be better informed and made more responsive to constantly changing business needs and opportunities.

The IT department cannot continue to dictate how IT-held data will be used: users have to be empowered to make these decisions themselves within their area of primary expertise. The focus of IT should instead be on making data available to end users, ensuring that it is not corrupted and that it can be audited when required, and on infrastructural issues like security, disaster recovery and so on. The job of the IT department should thus go far beyond merely distributing preconfigured reports, to encompass the more essential task of empowering users to get what they really need, when and how they need it, and to make sure that adequate resources are in place to get the job done. In short, IT departments need to work in a similar way to utility companies, providing users with infrastructure options but not with final solutions – and as with a power company, for example, fair and appropriate charge-out policies and procedures need to be put in place, so that IT can be transformed from a cost centre into a profit centre.

So, the primary role of IT should be to provide unfettered access to data, to make sure that the data will not be lost for any reason (even natural disasters), and to meet various levels of demand. While security has traditionally been another key issue for IT, even this area might be better handled by a specialized department, with IT providing assistance as needed. In any event, IT should not be able to prevent anyone with the appropriate security levels (which in some cases may exceed those of the head of IT) from getting to the data they need. Often, invocations of security or the impact on performance for “other system users” are simply a ruse to cover-up for the fact that the data is not readily available or that the system is not designed to cope with the demand.

We need to understand that a functional Business Intelligence (BI) system is governed by the same rules that govern any other large architectural undertaking like an office building or an elevated highway. It must meet the diversified needs of the community it serves and those of the people whose lives it affects. It also has to fit within the constraints already in place in the environment in which it is to be established – since very few of us get to work with a “clean slate”, and the legal and administrative hurdles can often be formidable.

Successful implementation of BI systems thus needs to be considered as part art and part science. Pure science has not been able to solve all society’s problems, and those who try to adopt a purely “scientific” approach to BI will not be successful. Fundamentally, the field of technology is not just about pursuing technical innovation, but about helping human beings become more effective and productive in pursuing their goals. In my following posts, I will be looking at how IT departments can go about designing systems that allow for just this sort of flexible, on-demand provision of information services to analysts, with the ultimate aim of enabling truly effective decision support for maximum repsonsiveness in today’s unpredictable business environment.

Arthur Ritchie

Business Intelligence: An Oxymoron?

Arthur — Tue, 24 Mar 2009 20:41:59 +0000

An old joke has it that the term “military intelligence” is an oxymoron – and in light of the current global financial crisis, it is tempting to put “Business Intelligence” in this category as well. Our inability to predict or deal competently with major events, from wars in the Middle East to the meltdown of global financial systems, shows just how ineffective our Business Intelligence/Data Warehousing strategies or “fit-for-purpose” reporting systems can be in responding to events as they unfold in this complex world. We are now confounded by the facts: we cannot predict the future; the largest military powers cannot conquer and control much weaker opponents; economists cannot adequately monitor essential financial systems. Automated trading systems, whose rules we once thought we understood and controlled, seem to have taken on a life of their own.

In my view, unless talented analysts are given unfettered access to whatever corporate data they require and the ability to analyze it as they see fit in the context of the many external data sources that are available, we will continue to find ourselves unprepared to deal with the unexpected. Implemented correctly, corporate Business Intelligence (BI) systems can support an organization’s best analysts as they challenge traditional business dogmas and develop a practicable way forward based on the facts, as recorded in detailed corporate data. To achieve this, IT departments need to stop acting as “data jailors” who strictly control which data will be accessible, in what form, and start empowering creative thinkers to realize their maximum potential, be it in marketing, manufacturing, distribution or some other field. In order to do this, however, IT departments need to start acting more like a power utility service: enabling “decision support” (to revive an older term for Business Intelligence) by providing corporate information or raw data as required, in the right amounts at the right time, while also serving as “consultants” who help end users access the data they require, when and how they need it.

In this series of blog posts, I will be discussing how an effective information infrastructure for decision support can be implemented without resorting to a “rip and replace” strategy that would involve completely scrapping the existing data warehouse. As the foundation of an optimal system for enterprise data management and decision support, I will be proposing the concept of Data Warehouse as a tiered architecture that combines three database models/technologies to support both standard reporting and “power analytics” as well as highly accessible storage of massive amounts of granular data to be used by the reporting and analytics engines.

Arthur Ritchie

Nearline 2.0 vs. the Archive

Arthur — Mon, 29 Sep 2008 16:54:09 +0000

In his most recent SAND blog post, Richard introduced the notion of “Nearline 2.0” and discussed how this concept, and related best practices, can be of vital importance to businesses dealing with the “data tsunami” we’ve been experiencing in recent years.

In this post, I’d like to step back a moment and explore the ways in which the dynamics of Nearline 2.0 differ from traditional methods of data archiving in terms of their approach to keeping data warehouse size under control.

Putting Your Database “on a Diet”

Faced with massive and continually increasing growth in data volumes, data warehouse administrators have come up with a number of techniques designed to maintain acceptable warehouse performance. These include pre-building aggregates and Key Performance Indicators (KPI’s) from large amounts of detailed transaction data, and indexing as many columns as possible in order to speed up query processing. As data warehouses continue to grow, however, the time required to do all the necessary preprocessing of data increases to the point where these tasks can no longer be performed within available “batch windows” when the warehouse is not being accessed by users. So, trade-offs need to be made. Doing less preprocessing work reduces the required time, but also means that queries that depend on aggregates, KPIs or additional indexes may take an inordinately long time to run, and may also severely degrade performance for other users as the system attempts to do the processing “on the fly”. This impasse leads to two possible choices: either stop providing the analytic functionality – making the system less valuable, and users more frustrated, — or “put the database on a diet” by moving some of the data it contains to another location.

Both Nearline 2.0 and archiving solutions can help trim down an over-expanded database: these allow substantial reduction of database size through implementation of an Information Lifecycle Management (ILM) approach, where unused or infrequently used detailed transactional data is removed from the online database and stored elsewhere. When the database is smaller, it will perform better and be capable of supporting a wider variety of user needs. Aggregates and KPI’s will be built from a much smaller amount of detailed transaction data. Additionally, column indexing will be more practicable as there will be fewer rows per column to be indexed. The natural side effect is, of course, that there is much less data to be analyzed and compared.

Getting “Lean” Not “Mean”

There are a number of important differences between archiving warehouse data (using products from Open Text, EMC Documentum, and so on) and storing it in Nearline 2.0 (using SAND/DNA). However, since both types of product are used to hold data that has been moved out of the main “online” system, it is unclear to some why one would need to be implemented if the other is in place. To help clarify why one or the other type of system (or both) might be required in a given situation, it is worthwhile to go over the major points of contrast between Nearline 2.0 data and archived data.

Nearline 2.0

In the Information Lifecycle Management approach, the Nearline 2.0 repository holds data that is used less frequently than the “hottest”, most current data, but which still needs to be readily available for analysis or for constructing new/ revised analytic objects for the warehouse to evaluate emerging trends.

While the exact proportion of Nearline 2.0 to online data will vary, the amount of “less frequently used” data that needs to be kept available is normally quite large. Moving this out of the main database greatly reduces the pressure on the online database and enables continued performance of standard database operations within available time windows, even in the face of the explosive data growth that many organizations are currently facing.

Thus, the archiving requirements described above do not apply to a Nearline 2.0 product such as SAND/DNA, which is designed to reduce the size of the online warehouse database, while at the same time keeping the data more or less transparently accessible to end users who may need to use it for analysis, for rebuilding KPI’s and so on.

2008 Season’s Greetings

admin — Tue, 11 Dec 2007 14:41:51 +0000

To all of our friends and associates,

As we approach the end of the year, I would like to express my appreciation for your interest in and support of SAND Technology during 2007.

This year was an exciting one for SAND, as we continued to gain recognition and market share for our SAND/DNA solutions, especially our nearline offering for SAP BI. On the product front, our development team has put extensive work into a range of exciting new analytical features to be included in the upcoming release 5 of SAND/DNA Analytics. We have also introduced two new offerings based on SAND/DNA Access technology, to provide very cost-effective solutions for organizations with specific requirements for nearline functionality.

Starting 2007 with the official certification of the SAND/DNA solution for use with the SAP NetWeaver BI 7.0 release, SAND has broadened its customer base worldwide with new customers in the financial services, consumer products and telecommunications industries, among others.We have continued to promote the SAND/DNA solution in the SAP world with well-received presentations at SAPPHIRE 07, SAP TechEd ‘07, and elsewhere. SAND personnel have also co-authored a number of articles appearing in prominent SAP-oriented journals such as SAP NetWeaver Magazine and SAP Info.

We continue to develop strong and mutually beneficial relationships with numerous channel partners around the world, and look forward to continued growth in the coming year.

All of us at SAND extend our best wishes for a very happy holiday season.

Arthur Ritchie

Moving Beyond the Data Warehouse Impasse – Part 3

Arthur — Wed, 10 Oct 2007 14:36:09 +0000

Once you extend your thinking beyond the data warehouse and “free” the data to speak for itself, the potential applications of the data un-warehouse concept are virtually unlimited. Let me suggest three powerful possible applications that would offer substantial benefit:

As a “Trust but Verify” engine for senior management to check the validity of conclusions resulting from analysis based on data warehouse information

As a “Sanity Check” component of a larger IT/Business Alignment process, an exercise now recommended by many IT consultants to ensure the synchronization of business objectives and IT actions

As a “Project De-risking” device applied to IT development projects, particularly Business Intelligence projects.

Trust but Verify

The notion of a “Trust but Verify” application derives from a concept promoted by President Ronald Reagan in the 1980’s as the US and the USSR undertook a nuclear disarmament process. The idea was to develop a process in which neither party would assume that the other party was lying, creating a sense of professional good will. Yet should a question arise, either party had the right to examine the details – the data of record – to verify the word of the other.

This practice is now applicable to issues of corporate governance and to situations where considerable investment in mergers and acquisitions is based upon analysis from data warehouse applications. Senior managers would now have the means (they have always had the right) to examine the data in the data store of record to verify the factual basis of proposals, to prevent a misreading of the situation and avoid the costs of mistaken action.

Sanity Check

The “Sanity Check” application relies on the ability of the original data to enable insight into the alignment of IT practice objectives with the business goals of the organization. The high cost of information technology and staffing has prompted firms like Gartner Group to develop an IT/Business Alignment process for determining to what extent, if at all, the process of data warehousing is leading an organization down a path that diverges from its corporate objectives, and a data store of record that supports full analytical access would provide an ideal tool for performing such an assessment.

For example, undue influence of the IT department on product choices, operational procedures and the technical organization of the data warehouse may be inadvertently masking the company’s vulnerability to changes in the market such as pricing increases, decreased demand or a shift in competitive practices. Using the data store of record as a source of information in “compare and contrast” exercises could help identify errors and suggest ways to correct the procedures and restore alignment with business objectives.

Project De-Risking

The “Project De-Risking” application would offer corporate management teams the opportunity to control one of the most volatile cost centers in business. For a variety of reasons, new Business Intelligence (BI) projects incur unique risks. Compared to a physical construction project, BI projects are built on an uncertain foundation, using untested materials, by workers without relevant experience, and no clear architectural plan. It is no wonder that failure rates are as high as they are. Yet, while de-risking BI projects is a critical goal, no standard industry best practice has been developed to address the issue. The data un-warehouse could answer this need by providing the data of record, untainted by assumptions of other projects that have come before. The data of record can be used as a test bed for the assumptions of the project – the equivalent of a control group in clinical trials.

Delivering Real Understanding

The idea of building a reference data repository alongside the data warehouse may sound radical, but the notion of a data warehouse was itself radical not that long ago. The concept of a parallel system that maintains a historical data store of record is as time-honored as the techniques of double-entry bookkeeping to maintain auditability and separation of responsibilities (e.g. Accounts Receivable versus Accounts Payable, General Ledger accounting versus internal audit versus external audit) in accounting.

New technologies have reduced costs, new thinking has extended the mandate of business intelligence, and new business requirements are driving new infrastructure developments. Perhaps the time has come to look towards a parallel data universe, one that offers the potential for a degree of business insight and project control that is beyond the scope of the traditional data warehouse.

Arthur Ritchie