Future of Data Warehousing

The Warehouse is supported in customers by careers and massive sunk dollar cost - so many will remain for a long time because too many people would lose face by removing it.

The Warehouse is pushed heavily by several of the most powerful technology companies on the planet; Because the warehouse can only run on their RDBMS platforms, and their share price is dependent on the number of database licenses they sell.
For them, any move away from the warehouse to a new technology will cannibalize their core revenues.

“Elephants Dance with Elephants” and so the big customers will always buy from the big vendors (who MUST sell RDBMS) - for a whole host of reasons that are nothing to do with the inappropriateness of the technology or the massive costs involved. They buy from the big vendors often because of PERCEIVED risk (everyone else has bought it, so my job is safe if I buy it).

Eventually new technologies will replace the warehouse. But not until all the stake holders in the massive warehouse projects have moved on to places where their careers cannot be hurt by the change.

Don’t get too hung up on the technology - it is hardly ever the reason people buy.

Livedashboards and Qlik view have huge draw backs

Livedashboard can at the most be called as a web replacement of excel and may not be able to cater to enterprise level needs
Qlik view in memory again neglects ETL process.
Bad data in is bad BI out
also requires complete up gradation on hardware to 64 bit machines and huge RAM upgrades
I don’t see any of these are really having any radical technological enhancements
Prashant Telang
www.transpacific.in

These BI tools really do look useful, however I don’t yet see how they make the DWH entirely obsolete.

The DWH does not just store data, but consolidates, normalizes, maintains history (that the source application might not do at all) as well. If you want to do meaningful analysis, you should have consistent data, so you need some kind of pre-processing and a store for the cleansed data. (except if you have a modern, fully on-line and synchronised architecture all the way, something I haven’t seen yet in any larger-scale enterprise).

As a Qlikview Reference I am a devotee of the tool and it is revolutionisng our business, but Peter is right the DWH has a - albeit small - role to play for the reasons described.

I guess it all depends on how you define “data warehouse” and where you use it. As a consultant to both E&C firms and O/O, I have to agree more with Peter. We use data warehouses to collect and cleanse data from one engineering application so that it can be shared with other. This helps us provide consistency and accuracy. It also provides the data history that we sometimes need. We also deal with data that is more rapidly changing and is not very static.

In the situations I deal with, engineering group “A” may enter data in their system but they are not ready to “release it” to the other groups yet. Status settings and “publishing” routines help us control that. Would something like that be available with these new systems? Maybe not, especially if the source system does not keep a history. How would these new systems find the last “issued” data if the user in the source application is working on a possible upgrade of a system in a plant or facility?

Now, I do think that these BI solutions have their place and sound great. I just am not convinced that they will do away with data warehouses.

data warehousing is a concept, and as such won’t be obsolete, at least any time soon. implementation of the concept, however, will change as technologies change. the need for standard data definitions, aggregation, and so on won’t go away, although every component in a DW implementation may take a different form over time. i do appreciate the need for a good sensational byline to attract readers ;)

Article about BI vs DWarehouse

We’re another BI software vendor who is helping to make data warehouses less necessary, with direct access to disparate data sources and pre-aggregration and data caching technologies.

But there will still be needs in large enterprises to use ETL technologies to cleanse and transform data and to use data warehouses to take care of the summarizations to deliver even higher performance.

So yes, new BI solutions can obviate data warehouses in some scenarios, but not kill them off everywhere.

Very educational. Thanks!

I wonder how the Gartner Quadrant for BI was built. PRELYTIS, the tool compared to QlikView in this report, is not included in the Gsrtner Report while QlikView was ranked in the top.

Manuel,

I can tell you that Gartner has a annual revenue requirement of $15M for consideration for the magic quadrant, so perhaps PRELYTIS is below that.

Yeah it’s not so long ago Qlikview wasn’t on there. I think the argument about data warehouses being dead is the wrong one to be having. Qlikview’s real benefit for me is time to value and agility - whether reading direct from data sources or a warehouse. Oh and cost and simplicity too, rather than trying to understand a crazy complex stack and what you need to buy you are delivering BI to your company after a few days consulting

We are also a vendor of a new style BI product, and it is possible to get the ETL, history, data cleansing benefits of the data warehouse plus the agility of development and end user simplicity offered by these new tools.
The best of both worlds at half the cost and half the effort.

Even Qliktech say that “In memory” is not their core differentiator, it is the structure of their data and cross referencing that is brilliant.

So if a new BI tool provided similar structures and cross referencing are available in large, scalable disk based data structures that perform well, then you have the same capabilities as DW plus the same benefits of QV.

That’s where we play, and it is definitely possible to combine the best of both worlds.

What this means also is that new generation BI like LiveDashBoard do not require a specific technology to be in place to deploy KPIs/business dashboards.
Companies’ IT infrastructures are constantly changing, moving away from traditional Windows + datamart/warehouse in place; new technologies come in place: open source, warehouses, and companies are expecting to pay less for the same. IT Heads and (budget holders) do not want to be stuck with one technology, costing always more. They are requesting for flexible solutions/inter-operable able to integrate with their current technologies, and even able to take into account their future technology investments.
I am not a specialist of QlikView but I have seen (press, news, national BI forums) that LiveDashBoard is being chosen over traditional BI (published by the Gartner) simply because it was working, because it required no or less development/scripting, leading to a lower TCO, and foremost because business users would get their KPIs, & business dashboards in a few weeks only, which, we shouldn’t forget, is the purpose of such a project.
Should we be listening to the needs of the market, or the marketing campaigns of traditional vendors?

[…] in this space. Recently, I had chance to read a very interesting write up made by TEC – Are Data Warehouses as Dead as the Dodo?, which is exploring a promising future of new BI technologies to replace data warehousing […]

Marion, the cynic in me says that until IT managers are motivated (paid?) to listen to the market, they will instead listen to the major vendors.
There is less risk to their jobs by authorising a $50Million SAP project that fails than there is in spending $50K on a PoC for a “Non Mainstream” product - whether it fails or succeeeds.

In fact the more they spend the less likely it is that anyone will ever say it failed.

Tools like QV and the other new products need to be business driven in order to break the stranglehold major vendors have on the IT manager’s safety net.

30 years ago Microsoft was that minor vendor, breaking IBM’s hold on the data centre & selling Office function to the business. Where are they now?

Many businesses now are again held back by their IT departments and their commitments to a few major vendors.
Time for a shake up - and it should come from the business.

[…] in this space. Recently, I had a chance to read a very interesting write up made by TEC – Are Data Warehouses as Dead as the Dodo?, which is exploring a promising future of new BI technologies to replace data warehousing […]

I’m a Qlikview customer and do not think it removes the need for a DW - see all comments above. Mainly because one benefit of Qlikview “in memory” is also a draw back if you have warehouses with Terabytes or even Petabytes of data. I think the technological game changer in the storage of cleansed, validated and history presserving data warehouse will be the emergence of Column based RDBMS with the improved IO profile they give over traditional row based RDBMS.

Qliktech and other analytical front ends can not retain all of the enterprise data. As long as the analysis uses the raw data that was initially loaded into them, they are great. If it is not there then a new process must be built to incorporate the missing data. If the selected data gets too large then performance suffers to the point where the system fails to meet business needs.
A system like Prelytis is really just the old Federated query system with a nice front end to simplify the use of multiple data sources. However,
1.It can not address the performance issue where the response to a query is determined by the slowest part of the federated system
2. It can do nothing about variable data quality across multiple sources
3. It can do nothing about rationalization of data across multiple sources