Decision Support for Users Who Don’t Know What They Don’t Know

Since the beginning of the computer era, system designers have struggled to reconcile conflicting aims of performance vs. functionality and maintainability vs. adaptability. In the case of Business Intelligence, there has been no less of a need for tradeoffs in order to deliver workable systems. However, BI system design has also typically been constrained even further by four fundamental realities:

• Users don’t know what they don’t know – they will always have difficulty articulating what they need, until they actually start working with the data.
• We in IT tend to want to implement relatively restrictive systems with well-defined acceptance criteria, in order to get buy-in from users. This protects us from backlash – and we know that users with looming deadlines will inevitably settle for less than they really require.
• Our IT training and experience has taught us that any system able to accommodate all the information that might be needed would be beyond our means – and even if it were affordable, it would be unlikely to perform adequately. As a result, we have gone to great lengths to determine the minimum amount of information required, then extracted that information and transformed it into the records that we retain.

However, it must be admitted that we almost never accurately judge what is “the minimum amount of information required” – and this typically leads to a series of costly attempts later on to find the missing data and retrofit it into our carefully designed data models.

• Generally, corporate executives have given up on the possibility of understanding the complexities of the computer world, and so have relinquished control of computer systems to the IT department. This lack of overarching leadership often results in a conflict situation, with the IT department facing various frustrated end-user groups who “don’t know what they don’t know”.

If IT departments are to make any progress in implementing Decision Support systems that really work, things need to change. We urgently need to approach the task of designing data warehouses in a spirit of humility and mutual co-operation. We must also acknowledge that some aspects of the resulting system may miss the mark – so we will need to build into the system a means of detecting and correcting problems, and for accepting user feedback and then efficiently incorporating the necessary adaptations into the system.

For a data warehouse environment to be of real value, it needs to be able to operate on the basis that users don’t know what they don’t know, by supporting the twists and turns of “incremental thinking” whereby they only gradually come to an understanding of their requirements. If a system designer comes right out and asks users what they want to know, they will typically be met with uncertainty. At this point, individuals anxious to promote their own way of thinking may step in with the assurance that they know what is really needed, while business experts who need to anticipate the effects of any number of potential scenarios from hurricanes to “credit crunches” often find themselves unable to make their voices heard. Locking in to a rigid, inflexible design on the basis of such inadequate input practically guarantees that the system will fail to deliver value over the long term.

In a Business Intelligence environment, incremental thinking usually involves “feeling one’s way” towards a goal by asking successive questions. Frequently, the approach will change – in terms of query strategies and the data sets we want to analyze – as insight is gained into the problem. Analysts work much like a detective will operate at the scene of a crime, picking up and examining various clues but not being able to link them together until the necessary understanding is developed. There are many spectacular examples of the enormous business value that can be derived from being able to ask any question that may arise in an analyst’s train of thought, particularly in highly dynamic activities like fuel hedging and currency trading. Success in these areas requires that plenty of historical details be available for unfettered and recursive access – allowing for the types of question that often provide the biggest business benefits.

However, it is obviously not necessary or even feasible to scan all the details every time a question needs to be answered. For this reason, a combination of traditional reporting and analysis architectures with a Nearline 2.0 repository (used to store massive amounts of detail data) can be the ideal solution. This architecture takes advantage of breakthroughs in database federation techniques and cheap, scalable SMP processors to create a massively parallel architecture that can easily scale to meet future needs, with little or no chance of technological obsolescence. In future posts I will look more closely at how such an architecture can be designed and implemented to enable high-performance, flexible decision support for business “at the speed of thought”.