For a data framework solution like Nearline 2.0, XAM provides the ability to store large amounts of structured data (for example, call detail records, application logs, web logs, syslogs, low-level manufacturing data, data exported from an enterprise data warehouse such as SAP BW, and so on) securely and efficiently to ensure compliance with data governance requirements. SAND/DNA Access can use XAM storage providers like EMC Centera as WORM (Write Once Read Many) devices, ensuring that stored data can not be modified. Furthermore, the Centera XAM compliance solution offers robust integrated business continuity and disaster recovery protection for structured data in SAND/DNA Access. In turn, SAND/DNA Access maintains massive volumes of structured data as content for the XAM storage provider, enabling a complete enterprise solution for data governance and compliance. SAND/DNA Access also brings high-performance query capability for structured data in the XAM storage provider, making it possible to conduct true electronic discovery (e-discovery) on this data. All these considerations contributed to SAND’s decision to implement support for the new XAM API.

We began development of XAM API support at the beginning of October 2008, and within a month were ready to execute our benchmark tests at the EMC Hopkinton Lab. These tests, which were fully supported by the EMC Centera team, allowed us to validate our XAM API implementation and to evaluate the performance of the data migration process (moving data from SAND/DNA Access to the Centera server) as well as query speeds. In both cases, observed performance was very impressive: using limited processing power (4 blade servers), we were able to achieve data migration speeds higher than 400 MB per second, and query speeds of close to 11 million records scanned per second! This shows that the new XAM functionality of SAND/DNA Access is more than up to the task of enabling existing Centera or XAM storage provider customers to extend their infrastructure to incorporate data compliance and governance regimes for massive amounts of structured data.

Let’s start at the beginning of the process. In a traditional data warehouse implementation, data is extracted from operational systems, then transformed and loaded into the data warehouse. Figure 1 shows this process, usually called ETL (Extract, Transform and Load) or sometimes ELT (Extract, Load and Transform), in the context of the Corporate Information Factory.

In previous posts, I presented Nearline 2.0 as a “database extension” that can reduce TCO, guarantee satisfaction of SLAs, and improve the performance of the online database that underpins the data warehouse solution. This configuration is based on the traditional concept of Information Lifecycle Management (ILM), normally implemented as a tactical effort to protect the enterprise against problems associated with the “data tsunami”. Figure 2 illustrates the incorporation of Nearline 2.0 into the data management framework as a database extension used to store (and maintain access to) older or less frequently used detail data.

While this ILM concept is a sound one, it has important limitations because of its reliance on migrating data from the online database to the nearline repository, with all the technical and human resource requirements this entails. Is it really necessary to do it this way? In my last post, I introduced the notion of Just-in-Case (JIC) data, which involves maintaining access to the detail data that underlies data warehouse summary tables. If the JIC data (which is static by definition) is “nearlined” at creation time — or as as soon as possible after it is created, — we can avoid the costs and system impact of moving it into the data warehouse. Why should we migrate JIC data to the online data warehouse, only to face the additional costs and hassles of archiving it later on? This will be a vital consideration when strategic decisions are being made about the enterprise data management framework.

It clearly makes much more sense for the data to be stored directly in the Nearline 2.0 repository. At SAND, we call this type of architecture, which is only possible because of the availability of Nearline 2.0 technology, the Corporate Information Memory (CIM). This is illustrated in figure 3.

The key benefit of this architecture for any enterprise is that it is based on an evolution of the current architecture, and so protects existing investments in people skills, software and equipment. This fits perfectly with Gartner’s observations in a recent report on “Key Issues for Delivering a Data Warehouse Project, 2008”:

“The big theme for data warehousing in 2008 is the increased demand for more data, in more places and doing so with an evolutionary approach. Data warehouses are mission-critical and integrated into operations. Failure to support business process change and inflexibility will not be tolerated. The problem is how to take the years of effort and funds invested previously and leverage them into a modern data warehouse because “rip and replace” strategies are not acceptable.”

So, it is not difficult to see how Nearline 2.0 can act as a pivotal element in an enterprise data management framework, and how it presents one of the best available solutions to the critical issues that have arisen in this area. In my next post, I will discuss the benefits of this type of architecture in more detail.

Richard Grondin

The Physical Table Partitioning Model

The first of these new data modeling options is based on physical table partitioning. The largest tables in a data warehouse or data mart can be physically divided between an online component and a Nearline counterpart. This allows the existing data model to be maintained, while introducing a “right-sizing” concept where only the regularly accessed data is kept online, and all data that doesn’t require such an expensive and/or hard to manage environment is put into the Nearline 2.0 solution. A typical rule of thumb for defining partition boundaries is based on the 90-day aging principle, so that any static data older than 90 days is migrated from the online warehouse to the Nearline 2.0 repository.

Now, many forms of enterprise data, such as CDR, POS, Web, Proxy or Log data, are static by definition, and are furthermore usually the main sources of data warehouse growth. This is very good news, because it means that as soon as the data is captured, it can be moved to the Nearline 2.0 repository (in fact, it is conceivable that this kind of data could be fed directly to Nearline 2.0 from the source system – but that is a topic for another post). Because of the large volumes involved, this kind of detail data has usually been aggregated at one or more levels in the enterprise data warehouse. Users generally query the summary table in order to identify trends, only drilling down into the details for a specific range of records when specific needs or opportunities are identified. This data access technique is well known, and has been in use for quite some time.

The Online Summary Table Model

This leads me to the second novel design option offered by Nearline 2.0: the ability to store all static detail data in the Nearline repository, and then use this as the basis for building online summary tables, with the ability to quickly drill to detail in the Nearline 2.0 repository when required. More specifically, the Nearline 2.0 solution can be used to feed the online system’s summary tables directly. The advantage of this implementation is that it substantially reduces the size of the online database, optimizes its performance, and permits trend analysis on even very long periods. This is particularly useful when looking for emerging trends (positive or negative) related to specific products or offerings, because it gives managers the chance to analyze and respond to issues and opportunities within a realistic time frame. A recent press release from Experian provides an excellent example of how business value can be produced by expert analysts using long-term historical data and the right set of tools.

Some organizations are already building this type of data hierarchy, using Data Marts or analytic cubes fed by the main Data Warehouse. I call this kind of architecture “data pipelining”. Nearline 2.0 can play an important role in such an implementation, since its repository can be shared between all the analytic platforms. This not only reduces data duplication, management/operational overhead, and requirements for additional hardware and software, it also relieves pressure on batch windows and lowers the risk of data being out of synch. Furthermore, this implementation can assist organizations with data governance and Master Data Management while improving overall data quality.

The Just-In-Case Data Model

Another important data modeling option offered by Nearline 2.0 relates to what we can call “just-in-case” data. In many cases, certain kinds of data will also be maintained outside the warehouse just in case an analyst requires ad hoc access to it for a specific study. Sometimes, for convenience, this “exceptional” data is stored in the data warehouse. However, keeping this data in an expensive storage and software environment, or even storing it on tape or inexpensive disks as independent files, can create a data management nightmare. At the same time, studies demonstrate that a very large portion of the costs associated with ad hoc analysis are concentrated in the data preparation phase. As part of this phase, the analyst needs to “shop” for the just-in-case data to be analyzed, meaning that he or she needs to find , “slice”, clean, transform and use it to build a temporary analytic platform, sometimes known as an Exploration Warehouse or “Exploration Mart.

Nearline 2.0 can play a very important role in such a scenario. Just-in-case data can be stored in the Nearline repository, and analysts can then query it directly using standard SQL-based front-end tools to extract, slice and prepare the data for analytic use. Since much less time is spent on data preparation, far more time is available for data analysis — and there is no impact on the performance of the main reporting system. This acceleration of the data preparation phase results from the availability of a central catalog describing all the available data. The Nearline repository can be used to directly feed the expert’s preferred analytic platform, generally resulting in a substantial improvement in analyst productivity. Analysts can focus on executing their analyses, and on bringing more value to the enterprise, rather than on struggling to get access to clean and reliable data.

At SAND, we have developed a new analytical offering based on this approach: “Database on Demand” or DBOD. Using DBOD, an analyst can specify their data requirements and have a 100% indexed RDBMS built directly from these specifications, ready for querying by front-end tools such as SAS, SPSS, Business Objects, MicroStrategy and so on. I’ll touch on that more in a future post.

Richard Grondin

First Steps

The first step in the Nearline 2.0 Quick Check involves installing a set of measurement tools to identify the overall size of each data warehouse and data mart used by the enterprise, along with the annual growth rate of each (note that it is not unusual for organizations to have more that one data warehouse in production at a given time, for a variety of reasons). A first set of metrics, generally extracted by the operational staff, provides a general analysis of storage usage over a specific period. The second set (pertaining to database objects) can be provided by the DBA team or collected using the Quick Check measurement tools. After discussions with key personnel, the collected metrics are used as the basis for estimating the TCO savings that would result from implementing various Nearline 2.0 scenarios.

The final set of metrics collected during the Nearline 2.0 Quick Check relates to organizational SLAs and TCO. Batch windows are analyzed to identify the major points of contention, and the processes that require long execution times and place a heavy load on processors, the network and the I/O subsystem. Frequently, the operational team will already know which processes are causing them nightmares, and can say whether they occur on a daily, weekly, monthly, quarterly or annual base. TCO can be more difficult to evaluate, and for this the reason needs to be examined with the assistance of organizational management.

The Nearline 2.0 Advantage

Of course, at the heart of this approach is the recently developed Nearline 2.0 concept, with all the new data management scenarios this has enabled. As I mentioned in a previous post, older Nearline 1.0 or archiving solutions presented major drawbacks, in that data removed from the online environment became very difficult and costly, if not impossible, to access. Because of this, determining the precise timing of data migration to a Nearline solution or Archive is a critical decision that can have a major impact on the organization.

A Nearline 2.0 solution, because of its performance characteristics and unique internal architecture, enables implementation of Nearline scenarios designed to deliver the highest return on investment and reduction of TCO, without giving up high-performance, flexible access to data for analytic purposes. Elaboration of the scenario doesn’t require any in-depth analysis of data access patterns. Rather, the data can be “nearlined” as soon as it becomes “static”, meaning that no more updates are planned for it. Analysis of update requirements will show that some data records are never updated after their creation, and can therefore be considered for nearlining right away — this is frequently the case with CDR, RFI or transaction log data. Data of this sort offers some very interesting data modeling options in a Nearline 2.0 solution, a topic I will be covering in my next post.

Quick Check Results

Once the Nearline 2.0 Quick Check is complete, a report can be prepared describing the current situation and where the enterprise will be in 6 months, in 1 year and so on. What is the expected cost to the enterprise of supporting the current rate of growth? What would be the effect of implementing various Nearline 2.0 scenarios? Armed with this information, the management team will have clear facts on which to base a decision to implement Nearline 2.0. And, based on my experience over the last few years, I’m pretty sure that they will decide to proceed earlier rather than later.

Richard Grondin

Cure or Prevention?

The “cure” type of implementation is typically associated with a data warehouse “rescue” project. This is undertaken when the data warehouse grows to a point where database size causes major performance problems and affects the warehouse’s ability to meet Service Level Agreements (SLAs). In these kinds of situation, it is mainly the operations division of the organization that is affected, and who demand an immediate fix that can take the form of a Nearline implementation. The question here is: How quickly can the “cure” implementation stabilize warehouse performance and ensure satisfaction of SLAs?

On the other hand, the best practice approach, much like current practices related to healthy living, focuses on prevention rather than on curing. In this respect, best practices dictate that the Nearline 2.0 implementation should start as soon as some of the data in the data warehouse becomes “infrequently accessed”. Normally, this means data older than 90 days, since the access rate for granular data older than 90 days is usually minimal. The main idea is to keep the size of the data warehouse from inflating for no good business reason, by nearlining data as soon as possible. Ultimately this should work to protect the enterprise from an operational crisis arising from deteriorating performance and unmet SLAs.

In order to better judge the impact of using either of these two approaches, it is important to understand the various steps involved in the “Data Nearlining” process. What do we find when we “dissect” the process of nearlining data?

Dissecting the “Data Nearlining” Process

“Data Nearlining” involves multiple processes, whose performance characteristics can significantly influence the speed at which data is migrated out of the online database. The various processes can be grouped into two major steps: data extraction and database housekeeping.

Data Extraction

• The first step (optional in some cases) is to lock the data that is targeted by the nearlining process, in order to ensure that the data is not modified while the process is going on.
• Next comes the extraction of the data to be migrated. This is usually achieved via an SQL statement based on business rules for data migration. Often, the extraction can be performed using multiple extraction/consumer processes working in parallel.
• The next step is to secure the newly extracted data, so that it is recoverable.
• Then, the integrity of the extracted data must be validated (normally by comparing it to its online counterpart).

Database Housekeeping

• Next, delete the online data that has been moved to nearline.
• Then, reorganize the tablespace of the deleted data.
• Finally, rebuild/reorganize the index associated with the online table from which data has been nearlined.

The Database Housekeeping process is often the slowest part of a Data Nearlining process, and thus can dictate the pace and scheduling of the implementation. In a production environment, the database housekeeping process is frequently decoupled from ongoing operations and performed over a weekend. It may be surprising to learn that deleting data can be a more expensive process than inserting it, but just ask an enterprise DBA about what is involved in deleting 1 TB from an Enterprise Data Warehouse and see what answer you get: for many, the task of fitting such a process into standard Batch Windows would be a nightmare.

So, it is easy to see that starting earlier in implementing Nearline 2.0 as a best practice can help to massively reduce not only the cost of the implementation, but also the time required to perform it. Therefore, the main recommendation to take away from this discussion is: Don’t wait too long to consider embarking on your Nearline 2.0 strategy!

That’s it for today. In my next post, I will take up the topic of which data should be initially considered as a candidate for migration

Nearline 2.0 enables you to keep all your valuable data available for analysis.

• A clearer understanding of emerging trends in the business – what will go well in the future as well as what is now “going south”
• Better support for iterative analyses, enabling more intensive Business Performance Management (BPM)
• Better insight into customer behavior over the long term
• More precise target marketing, bringing a three- to five-fold improvement in campaign yield

Nearline 2.0 enables you to dramatically increase information storage and maintain service levels without increasing costs or administration requirements.

• Extremely high compression rates give the ability to store considerably more information in a given hardware configuration
• A substantially reduced data footprint means much faster data processing, enabling effective satisfaction of Service Level Agreements without extensive investments in processing power
• Minimal administration requirements bring reductions in resource costs, and ensure that valuable IT and business resources will not be diverted from important tasks just to manage and maintain the Nearline 2.0 implementation
• High data compression also substantially reduces the cost of maintaining a data center by reducing requirements for floor space, air conditioning and so on.

Nearline 2.0 simplifies and accelerates Disaster Recovery scenarios.

Nearline 2.0 keeps all detail data in an immutable form, available for delivery on request.

Having read-only detail data available on-demand enables quick response to audit requests, avoiding the possibility of costly penal-ties for non-compliance. Optional security packages can be used to control user access to the data.

Nearline 2.0 makes it easy to offload data from the online database before making final decisions about what is to be moved to an archiving solution.

The traditional archiving process typically involves extensive analysis of data usage patterns in order to determine what should be moved to relatively inaccessible archival storage. With a Nearline 2.0 solution, it’s a simple matter to move large amounts of data out of the online database — thereby improving performance and guaranteeing satisfaction of SLA’s, — while still keeping the data avail-able for access when required. Data that is determined to be no longer used, but which still needs to be kept around to comply with data retention policies or regulations, can then be easily moved into an archiving solution. For more consideration of the differences between Nearline 2.0 and traditional archiving, see Arthur’s recent blog post.

Taken together, these benefits make a strong case for implementing a Nearline 2.0 solution when the data tsunami threatens to overwhelm the enterprise data warehouse. In future posts, I will be investigating each of these in more detail.

Richard Grondin

In this post, I’d like to step back a moment and explore the ways in which the dynamics of Nearline 2.0 differ from traditional methods of data archiving in terms of their approach to keeping data warehouse size under control.

Putting Your Database “on a Diet”

Both Nearline 2.0 and archiving solutions can help trim down an over-expanded database: these allow substantial reduction of database size through implementation of an Information Lifecycle Management (ILM) approach, where unused or infrequently used detailed transactional data is removed from the online database and stored elsewhere. When the database is smaller, it will perform better and be capable of supporting a wider variety of user needs. Aggregates and KPI’s will be built from a much smaller amount of detailed transaction data. Additionally, column indexing will be more practicable as there will be fewer rows per column to be indexed. The natural side effect is, of course, that there is much less data to be analyzed and compared.

Getting “Lean” Not “Mean”

There are a number of important differences between archiving warehouse data (using products from Open Text, EMC Documentum, and so on) and storing it in Nearline 2.0 (using SAND/DNA). However, since both types of product are used to hold data that has been moved out of the main “online” system, it is unclear to some why one would need to be implemented if the other is in place. To help clarify why one or the other type of system (or both) might be required in a given situation, it is worthwhile to go over the major points of contrast between Nearline 2.0 data and archived data.

Archive

Normally, the concept of electronic archiving focuses on the preservation of documents or data in a form that has some sort of certifiable integrity (for example, conformity to legal requirements), is immune to unauthorized access and tampering, and is easily subject to certain record management operations within a defined process – for example, automatic deletion after a certain period, or retrieval when requested by an auditor. The archive is in fact a kind of operational system for processing documents/data that are no longer in active use.

The notion of archiving has traditionally focused on unstructured data in the form of documents, but similar concepts can be applied to structured data in the warehouse. An archive for SAP BI, for example, would preserve warehouse data that is no longer needed for analytical use but which needs to be kept around because it may be required by auditors, as would be the case if SAP BI data were used as the basis for financial statements. The archive data does not need to be directly accessible to the user community, just locatable and retrievable in case it is required for inspection or verification – not for analysis in the usual sense. In fact, because much of the data that needs to be preserved in the archive is fairly sensitive (for example, detailed financial data), the ability to access it may need to be strictly regulated.

While many vendors of archiving solutions stress the performance benefits of reducing the amount of data in the online database, accessing the archived data is a complicated and relatively slow process, since it will need to be located and then restored into the online database, or accessed directly in a much slower backup data base that is not readily maintained from a performance or accessibility perspective. For this reason, it is unrealistic to expect archived data to be usable for analysis/reporting purposes.

Nearline 2.0

In the Information Lifecycle Management approach, the Nearline 2.0 repository holds data that is used less frequently than the “hottest”, most current data, but which still needs to be readily available for analysis or for constructing new/ revised analytic objects for the warehouse to evaluate emerging trends.

While the exact proportion of Nearline 2.0 to online data will vary, the amount of “less frequently used” data that needs to be kept available is normally quite large. Moving this out of the main database greatly reduces the pressure on the online database and enables continued performance of standard database operations within available time windows, even in the face of the explosive data growth that many organizations are currently facing.

Thus, the archiving requirements described above do not apply to a Nearline 2.0 product such as SAND/DNA, which is designed to reduce the size of the online warehouse database, while at the same time keeping the data more or less transparently accessible to end users who may need to use it for analysis, for rebuilding KPI’s and so on.

Nearline 2.0, by contrast, allows historical data to be accessed with near-online speeds, empowering business analysts to measure and perfect key business initiatives through analysis of actual historical details. In other words, Nearline 2.0 gives you all the data you want, when and how you want it. (And without impacting the performance of existing warehouse reporting systems!)

Aside from the obvious economic and environmental benefits of this software-centric approach and the associated best practices, the value of Nearline 2.0 can be assessed in terms of the core proposition cited by Tim O’Reilly when he coined the term “Web 2.0″:

“The value of the software is proportional to the scale and dynamism of the data it helps to manage.”

In this regard, Nearline 2.0 provides a number of important advantages over prior methodologies:

Keeps data accessible: Nearline 2.0 enables optimal performance from the online database while keeping all data easily accessible. This massively reduces the work required to identify, access and restore archived data, while minimizing the performance hit involved in doing so in a production environment.

Keeps the online database “lean”: Because Nearline 2.0 data can still be easily accessed by users at near-online speeds, it allows for much more recent data to be moved out of the online system than would be possible with archiving. This results in far better online system performance and greater flexibility to further support user requirements without performance trade-offs.

Relieves data management stress: Data can be moved to Nearline 2.0 without the substantial ongoing analysis of user access patterns that is usually required by archiving products. The process is typically based on a rule as simple as “move all data older than x months from the ten largest tables”.

Mitigates administrative risk: Unlike archived data, Nearline 2.0 data requires little or no additional ongoing administration, and no additional administrative intervention is required to access it.

Lets analysts be analysts: With Nearline 2.0, far less time is taken up in gaining access to key data and “cleansing it”, so much more time can be spent performing “what if” scenarios before recommending a course of action for the company. This improves not only the productivity but also the quality of work of key business analysts and statistical gurus.

Copes with data structure changes: Nearline 2.0 can easily deal with data model changes, making it possible to query data structured according to an older model alongside current data. With archive data, this would require considerable administrative work.

Leverages existing storage environments: Compared to older archiving products/strategies, the high degree of compression offered by nearline 2.0 greatly increases the amount of information that can be stored as well as the speed at which it can be accessed.

Keeps data private and secure: Nearline 2.0 has optional privacy and security packages that protect key information from being seen by ad-hoc business analysts (for example: names, social security numbers, credit card information).

In short, Nearline 2.0 offers a significant advantage over older Nearline 1.0 and archiving technologies. When data needs be removed from the online database in order to improve performance, but still needs to be readily accessible by users to conduct long-term analyses or to rebuild aggregates/KPIs/InfoCubes for period-over-period analysis, Nearline 2.0 is currently the only workable solution available.

In my next post, I’ll discuss more specifically how implementing a Nearline 2.0 solution can benefit both your data warehouse and your business.

Richard Grondin