The Lowdown On Agile Data Warehousing

Many believe that 2015 was the year of Agile Data Warehousing and Business Intelligence. The traditional ways of doing things and the strategies  for data warehousing (DW) and Business Intelligence (BI) have been challenged. Agile strategies provide a better time to market, improve the satisfaction of stakeholders, enhance quality levels and ensure higher ROI as compared to traditional processes and strategies. With these factors in mind, there are also questions that need to be answered in order to cast down the doubts of many businesses and organisations – how do you make a modern data warehousing model fir any enterprise analytics strategy? How does the concept of agile development apply to data warehousing? What benefits can we expect from agile data warehousing? Where will the strongest ROI come from?

The established DW/BI community has started to accept agile changes to their existing, traditional infrastructure. Many experts believe that within this year and the coming years, there will be blog posts and case studies that will describe many user experiences with agile data warehousing.

Image credit:
Image credit:

Agile Data Warehousing Defined

The Agile approach is quickly gaining popularity as it solves many data warehousing issues like cost, low user adoption, the ever-changing requirements of businesses and the inability to quickly adapt to changes in technology and business markets.

Data Warehousing is defined as a systems design and implementation approach focused on delivering business values and functionalities early and on a continuous basis. Agile DW uses a framework that comprises specific design methods, testing applications, agile development and agile project methods combined to allow for implementation automation. It can also be defined as a database that contains homogenised and integrated information from one or more sources brought together in support of reporting and analysis. The said processes can be OLTP or online transaction processing systems like sales, accounting, finance, payroll, supply chain or marketing. It can also be used for external sources like supplier files or purchased marketing lists and social media sites.

Why Use Agile Data Warehousing?

There are several factors why agile data warehousing is the best option in building analytical databases. Let’s discuss two of the most important factors:

1. The Business-driven Approach

Building an analytical database is time consuming, complex and often pretty expensive, specifically when traditional data-driven methods are used. One fundamental reality of BI and DW is the imagination and data integration account for about 70%-80% of the project budget. It’s also not uncommon to want to integrate and homogenise most of the data before the first reports or queries are written, thus 70%-80% of the budget will be used before any business value is realised. Some of the field integration also takes about 1 year to complete. Thus, one of the first things you need to do is lower the amount spent on the data integration and homogenisation.

2. Risk Reduction and System Creation With High Adoption Rates

Business organisations that use traditional, all-or-nothing methods in data warehousing and business intelligence build unnecessary risk and eventually they will find out that they have created solutions that do not satisfy the organisation’s needs. One of the possible primary reasons is that business needs and priorities have shifted between the requirements when the system was first defined and when the analytical database was actually implemented. If agile methods are used or applied, values can be shown on a regular basis. The key tasks of the database design like data quality remediation and the data integration and homogenisation are broken into short delivery cycles that commonly last 2-4 weeks. These data-driven and data-focused stategiesare usually paired with Business Intelligence prototyping which allows the business to interact multiple times which ensures the analytical database contains the most useful information.

Image credit:
Image credit:

Agile Data Warehousing Vs Traditional Methods

There are a few differences between an agile approach and traditional data warehousing.

  1. The agile teams will usually follow usage-based strategy instead of data-driven approaches. DW/BI teams will be able to deliver valuable functionality that addresses required needs of the end users if they focus on how people will use the data warehouses.
  2. Agile data warehousing teams will deliver in increments, avoiding the risks that are associated with a big-bang release strategy.
  3. A disciplined agile DW/BI teams take a test-driven approach which leads to greater quality and dependable project management.

Agile data warehousing will give a high ROI if it’s applied to projects that require a hierarchical progression. Try applying it to a project that is mostly data access and then do it again for any project that is mostly reporting. For agile data warehousing to truly flourish, IT teams should learn to work in a collaborative and simplified manner with business customers.

Image credit:
Image credit:

Agile data warehousing methods can be applied in building analytical databases and project spending will be minimised if the work associated with designing and integrating is driven from the business side. Homogenisation and data integration will surely meet the organisation’s needs for actionable information.


Please enter your comment!
Please enter your name here