Agile BI and data warehouses

29 Aug

I have done a lot of data conversion projects and a lot of building of reports on financial information. But when participants in my courses ask about data warehouses my advice is a little more limited.

I usually tell people that data warehouses sound like a good idea and that every company should have one. People usually thank me politely for this but also advise me that they already have one – in fact they have around 17.

At that point we agree that if you use a cowboy based agile approach with no architecture you will add a lot of access databases and tactical databases to the existing chaos without simplifying the environment.  So we agree that we should do a substantial amount of careful design and strategy work before buildng too many databases (ie do up front architecture).

But then one of the group usually reveals that their company also did this and their architects are still trying to convince people that it is a good idea to build $100 million in infrastructure before seeing any benefits – and meanwhile projects are deciding they can’t wait for the utopian dream, so building “tactical datamarts”.

These tactical datamarts generally duplicate both data and interfaces without decommissioning, consolidating or even fixing existing data and interfaces.

So we end up with what is called spagettification driven development.

Even with my limited exposure the business intelligence and data warehouses I can see that this is both the wrong thing to do and an easy trap to fall into.  So a better approach would be to merge detailed architecture (upfront and ongoing) with a continuous delivery of value (new useful reports, simplified data and decommissioned redundant interfaces).

With that in mind I thought I would publish some of the links that participants have sent me to research this further:

An ebook on “agile data warehousing ” that several participants recommended (it will cost about $10)

A RUP based approach published by Scott Ambler

Cutter consorium page on business intelligence

Let me know if you have other links, or better yet some experience in successfully working with the implementation of data warehouses (using agile or not).

Posted by James King

Leave a comment

Posted by on August 29, 2010 in Agile, Design Process


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: