Posted by: Peter Scott | March 9, 2008

Data Warehouses are not dead, yet

One of the reasons that I left my old job to work with Rittman Mead Consulting was to get back to technology. Don’t get this wrong, I really did enjoy managing people and having responsibility for a BI practice, but that was at the expense of involvement in a lot of the delivery of systems; for every one day of project work, I tended to spend eight doing “other stuff”, and I was getting to miss the buzz of being at the sharp end of a project, well maybe not missing the 2:30am coding whilst eating a slice of pizza stuff (which is a young person’s game)

Recently, I have been set the challenge to grow the amount of data warehouse and data quality work I am involved in and perhaps at the same time help dispel the perception that Rittman Mead Consulting just do short engagements and training; in the three months or so I have been here I have spent just three days delivering training and 2.5 months on-site with a single customer developing a data warehouse, so that notion is a bit of a myth anyway. Maybe the new Rittman Mead web site will feature some of the longer term type of engagements we are involved in.

I suppose a good question to start with is “do people still develop data warehouses?” Mark brought up a similar idea the other week in his ‘future BI architecture’ piece which also attracted some good comments. Indirectly, I mentioned alternatives to DWs in my article on Real Time BI for the Evaluation Centre – I hope to either post the article here on the blog or perhaps on the Rittman Mead web site in the next few days. A lot of organisations have already invested in data warehouses, these will continue to need attention, either from the support perspective or for enhancement as new classes of information are added. Data warehouses still have a place in BI as way of facilitating the delivery of quality data to the reporting layer in an efficient way so I can’t quite see the death of the Data Warehouse just yet.

About these ads

Responses

  1. What is this obssession with the “death of such and such technology” of late?

    Does anyone honestly believe the acronym soup that characterizes the web2 architecture is seriously going to replace ANYTHING?

    Because I have yet to see proof it even works, let alone replace anything….

  2. Noons – I doubt that data warehouses will go in my working lifetime (but then I am old… ;-) ) so I will still have a job for some while!

    Maybe, we should really be speaking of evolution – DWs will change, become less monolithic, more federated, and support more types of data for query (such as image, free text – but getting the dimensionality of a photo defined is going to be a challenge!) And as ever the problem for people like us is to move shed loads of data quickly enough to be useful, and that is always going to be down database infrastructure and not some whizzy web query tool thing

  3. Well, they just announced quantum memories now. Apparently, something like a TB in less than 1/8″square, non-volatile, with the same read/write times of DRAM.

    I have difficulty even imagining the possibilities but if it becomes mainstream in say, 10 years, then it’s not widespread storing of photos I’m worried about: it’s full on HD video! :-)

    Worse: how do we index and x-ref all that so it can be queried usefully and come back in a usable time? Nope, classic indexing is not viable, we just can’t say “go build an index on those 3HB of videos”: our children will be able to use the index at current CPU and sort algorithm speeds…

    I think Jim Gray’s “Personal Petabyte” is right on schedule. Once more he is proven right: we have to prebuild the catalogues and indexes and then just populate them as needed. So much for “no need for data modelling and logical db design”.

    Darn, why did he have to go sailing!…

  4. I don’t see data warehouses dying anytime soon either. In fact, I am currently involved in a brand new project slated to be complete some time next year.

    One thing that might be changing is the idea of “real time” data warehousing. “Right time” is a better phrase, but even that seems too ambiguous. Certainly SOA is the way to go for real-time reporting and analysis (which is often the arena for managers stuck in their departmental islands in need of cross business process reports).

    I disagree with you on this point: “Data warehouses still have a place in BI as way of facilitating the delivery of quality data to the reporting layer”. While reporting is a huge part of presentation, I have worked on nothing but intense analytic projects designed to present upper management with competitive advantage. Usually this is in the form of complex applications, scoring engines, and the like.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: