• Hardware:
  • Cheap but powerful SMP hardware
  • Reasonably priced, fast and efficient SAN devices
  • Reasonably priced, fast and effective networks and switches for linking multiple SMP boxes together.
• Architecture:
  • Nearline data storage (not archiving) to hold the detail data used to build and maintain aggregates and cubes. There is a very clear differentiation between the Nearline 2.0 and archiving approaches to data and performance management.
  • Database federation techniques enabling reduction of the amount of data in key RDBMS “hot spots” and minimization of batch window requirements
  • Well-defined uses for aggregation, indexing, and MOLAP cubes to support reporting.
• Software:
  • Column-based data management technologies for avanced analytics, offering:
    • The ability to change data models “on the fly” to meet emerging requirements
    • Support for very wide tables (tens of thousands of columns), enabling very large numbers of KPIs
    • The ability to add new data types “on the fly”
    • The ability to allow existing applications to continue to work as data models change over time
    • The ability to present all available data in a simple, easily usable format, eliminating the need for analysts to navigate complex data relationships and slowly changing dimension constructs.
  • Better exploitation of new hardware and architectural techniques by traditional RDBMS systems
  • The ability to support massively parallel operations, allowing users to iteratively run very complex queries to find patterns in the data without having their thought process interrupted, without impacting other power users and other reporting users, and without interfering with the ability to meet critical SLAs.

Creating a “corporate memory” of this sort brings a number of major benefits:

• If it turns out that any or all of the original data is required for a new project (involving new aggregations, “cubes” or KPIs, for example), it is substantially less expensive and much easier to access it.
• If it becomes necessary to assess the value of the corporate data, this can be done carefully over a period of time using data mining techniques, rather than in a rush to meet specific project deadlines.
• The original detail data will only need to be handled once, and then can be retained as a permanent record for use in auditing the data warehouse if this becomes necessary (to prove compliance, to aid in conflict resolution, or simply to rectify “human errors” that were made at some point in the process)
• It becomes possible to document in the metadata layer who accessed the information, when and for what reason – incidentally, this can also provide valuable insight into the abilities of those who are responsible for manipulating the raw data.

In my next post, I’ll cover the ingredients for building Corporate Memory into a Next-Generation Data Warehouse.

• Users don’t know what they don’t know – they will always have difficulty articulating what they need, until they actually start working with the data.
• We in IT tend to want to implement relatively restrictive systems with well-defined acceptance criteria, in order to get buy-in from users. This protects us from backlash – and we know that users with looming deadlines will inevitably settle for less than they really require.
• Our IT training and experience has taught us that any system able to accommodate all the information that might be needed would be beyond our means – and even if it were affordable, it would be unlikely to perform adequately. As a result, we have gone to great lengths to determine the minimum amount of information required, then extracted that information and transformed it into the records that we retain.

However, it must be admitted that we almost never accurately judge what is “the minimum amount of information required” – and this typically leads to a series of costly attempts later on to find the missing data and retrofit it into our carefully designed data models.

• Generally, corporate executives have given up on the possibility of understanding the complexities of the computer world, and so have relinquished control of computer systems to the IT department. This lack of overarching leadership often results in a conflict situation, with the IT department facing various frustrated end-user groups who “don’t know what they don’t know”.

If IT departments are to make any progress in implementing Decision Support systems that really work, things need to change. We urgently need to approach the task of designing data warehouses in a spirit of humility and mutual co-operation. We must also acknowledge that some aspects of the resulting system may miss the mark – so we will need to build into the system a means of detecting and correcting problems, and for accepting user feedback and then efficiently incorporating the necessary adaptations into the system.

For a data warehouse environment to be of real value, it needs to be able to operate on the basis that users don’t know what they don’t know, by supporting the twists and turns of “incremental thinking” whereby they only gradually come to an understanding of their requirements. If a system designer comes right out and asks users what they want to know, they will typically be met with uncertainty. At this point, individuals anxious to promote their own way of thinking may step in with the assurance that they know what is really needed, while business experts who need to anticipate the effects of any number of potential scenarios from hurricanes to “credit crunches” often find themselves unable to make their voices heard. Locking in to a rigid, inflexible design on the basis of such inadequate input practically guarantees that the system will fail to deliver value over the long term.

In a Business Intelligence environment, incremental thinking usually involves “feeling one’s way” towards a goal by asking successive questions. Frequently, the approach will change – in terms of query strategies and the data sets we want to analyze – as insight is gained into the problem. Analysts work much like a detective will operate at the scene of a crime, picking up and examining various clues but not being able to link them together until the necessary understanding is developed. There are many spectacular examples of the enormous business value that can be derived from being able to ask any question that may arise in an analyst’s train of thought, particularly in highly dynamic activities like fuel hedging and currency trading. Success in these areas requires that plenty of historical details be available for unfettered and recursive access – allowing for the types of question that often provide the biggest business benefits.

However, it is obviously not necessary or even feasible to scan all the details every time a question needs to be answered. For this reason, a combination of traditional reporting and analysis architectures with a Nearline 2.0 repository (used to store massive amounts of detail data) can be the ideal solution. This architecture takes advantage of breakthroughs in database federation techniques and cheap, scalable SMP processors to create a massively parallel architecture that can easily scale to meet future needs, with little or no chance of technological obsolescence. In future posts I will look more closely at how such an architecture can be designed and implemented to enable high-performance, flexible decision support for business “at the speed of thought”.

The IT department cannot continue to dictate how IT-held data will be used: users have to be empowered to make these decisions themselves within their area of primary expertise. The focus of IT should instead be on making data available to end users, ensuring that it is not corrupted and that it can be audited when required, and on infrastructural issues like security, disaster recovery and so on. The job of the IT department should thus go far beyond merely distributing preconfigured reports, to encompass the more essential task of empowering users to get what they really need, when and how they need it, and to make sure that adequate resources are in place to get the job done. In short, IT departments need to work in a similar way to utility companies, providing users with infrastructure options but not with final solutions – and as with a power company, for example, fair and appropriate charge-out policies and procedures need to be put in place, so that IT can be transformed from a cost centre into a profit centre.

So, the primary role of IT should be to provide unfettered access to data, to make sure that the data will not be lost for any reason (even natural disasters), and to meet various levels of demand. While security has traditionally been another key issue for IT, even this area might be better handled by a specialized department, with IT providing assistance as needed. In any event, IT should not be able to prevent anyone with the appropriate security levels (which in some cases may exceed those of the head of IT) from getting to the data they need. Often, invocations of security or the impact on performance for “other system users” are simply a ruse to cover-up for the fact that the data is not readily available or that the system is not designed to cope with the demand.

We need to understand that a functional Business Intelligence (BI) system is governed by the same rules that govern any other large architectural undertaking like an office building or an elevated highway. It must meet the diversified needs of the community it serves and those of the people whose lives it affects. It also has to fit within the constraints already in place in the environment in which it is to be established – since very few of us get to work with a “clean slate”, and the legal and administrative hurdles can often be formidable.

Successful implementation of BI systems thus needs to be considered as part art and part science. Pure science has not been able to solve all society’s problems, and those who try to adopt a purely “scientific” approach to BI will not be successful. Fundamentally, the field of technology is not just about pursuing technical innovation, but about helping human beings become more effective and productive in pursuing their goals. In my following posts, I will be looking at how IT departments can go about designing systems that allow for just this sort of flexible, on-demand provision of information services to analysts, with the ultimate aim of enabling truly effective decision support for maximum repsonsiveness in today’s unpredictable business environment.

Arthur Ritchie

In this series of blog posts, I will be discussing how an effective information infrastructure for decision support can be implemented without resorting to a “rip and replace” strategy that would involve completely scrapping the existing data warehouse. As the foundation of an optimal system for enterprise data management and decision support, I will be proposing the concept of Data Warehouse as a tiered architecture that combines three database models/technologies to support both standard reporting and “power analytics” as well as highly accessible storage of massive amounts of granular data to be used by the reporting and analytics engines.

Arthur Ritchie