Although information analysis continues to be vital for insights into a company’s health and future, the traditional data warehouse structure may already be past its best-before date, due to major and radical transformations in the market.
It may just be a matter of “Evolve or Die”. . . Indeed, in my last blog post Are Data Warehouses as Dead as the Dodo? (LiveDashBoard and QlikView), I talked about how some software providers are dramatically changing the traditional way we design and deploy a data warehouse solution. I specifically reviewed two interesting solutions available in the market (LiveDashboard [Evaluate the product] and QlikView [Evaluate the product]). Having a keen interest on the topic and seeing the richness of options available for technology applications, I’ve decided to expand upon this by discussing several other options currently in the market. Certainly, at the end of the day, we will be seeing more and more innovative ways to analyze large sets of data. . .
Despite the various options to design and deploy data warehousing solutions, there appear to be a couple of characteristic features in the way most innovative vendors are addressing this space. One refers to database transparency, which means that the database is exploited by the user without an awareness of the technical infrastructure that supports it (connection to data sources, language programming complexity, etc.), and the other is high performance to analyze large sets of data in just a few minutes or even seconds. Most software providers are paving the way to new avenues by taking advantage of different technology alternatives, such as parallel processing capabilities and in-memory technologies, to name a few.
The two other players in the market that could bring a solution to your table and help you with your common data management and data analysis tasks are 1010data, with its innovative combination of technologies, and Illuminate, with its unique correlation database offering.
1010data, a software company based in New York, NY, has an interesting approach to providing data management and analysis services to its customers. Using a transparent database approach to solving the problem of performing “big data” analysis tasks, 1010data’s application provides an easy-to-use interface to analyze billions of rows of information from disparate sources, with high performance and advanced analytic capabilities. The 1010data product can be deployed as an appliance and as Software as a Service (SaaS), or can even be more specific as a Data Warehouse as a Service (DWaaS) solution—giving 1010data the flexibility to cover a wide range of potential customers in search for a data warehouse and analysis solution.
To perform data management tasks, 1010data uses an approach called ELTar (Extract, Load, and Transform as required), in which the transformation process is done as required by the user. As there is no design conception of workflows or series of tasks to collect data from disparate sources, users have the freedom to collect raw data and apply full analysis and transformation capabilities directly without having to wait for all the traditional collect-clean-transform-analyze cycle.
Some of the features and technologies that 1010data relies upon to gain performance and reliability are as follows:
• A columnar structure—1010data has a columnar structure, which enables the analysis of large volumes of data faster than traditional relational database designs
• Dynamic in-memory capabilities—1010data has an in-memory engine, which not only allows for improvements in data analysis speed, but also enables the swapping of data between the memory and hard disk, as needed
• Parallel and distributed processes capabilities—1010data ensures fast performance by distributing jobs to be executed in parallel as well as providing an architecture that can assign dedicated processes for every user, thereby avoiding problems associated with concurrent use
1010data is changing the traditional schema for a BI solution, and it’s reinforcing the data management structure with a proprietary database management system called TenBase. Finally, 1010data completes the application with a friendly user interface called 1010Vision, which allows users with no technical background to easily extract, prepare, explore, as well as analyze information using a graphical interface. 1010data intends to give users freedom to analyze data spontaneously, along with a more guided and friendly user experience.
1010data’s product strategy relies on giving the data management layer enough power to improve performance and allow users to take advantage of its proprietary technology as well as use the browser application along with more traditional BI solutions.
Illuminate is a company headquartered in Barcelona, Spain. Founded in 2005 and currently with offices in both Europe and North America, Illuminate has created a proprietary database management system, called correlation database management system (CDBMS), that enables it to help organizations build corporate data warehouses by applying a new and radical approach to modeling and deploying data management solutions for analytical purposes. In addition to its correlation database, Illuminate relies on the iSTORE product to exploit the benefits of the database and enable users to perform exploration and ad-hoc queries for analysis. All this is done using Illuminate’s proprietary technology, called VBS Technology, and an indexing system that enables correlating information as well as accelerating analysis and exploration. Illuminate serves a wide range of industries, including banking and health care, and its customers include companies such as Zurich, BBVA, and emagister.com.
One of the main features of the iSTORE native CDBMS is that all data are stored under a single layer, nicely differentiating it from other databases, such as column-oriented and online analytical processing (OLAP) cubes or star schemas. This single layer of information is related in such a way as to enable fast exploration and query association by the use of specific data indexes to improve information recovery and analysis of large volumes of data.
Some of the functional features of iSTORE are as follows:
• It can enable users to perform a true ad-hoc analysis. The universal indexing allows analysis across n dimensions of existing data sets.
• It can perform what Illuminate defines as “incremental queries.” This means that the user can start by asking a simple question, which will then be correlated to subsequent questions, so that users will be able to refine or expand the scope of information as desired. The incremental query process allows users to build the queries that they deem appropriate based on the issues at hand.
• It can “cluster” or group information by specific criteria on the fly, enabling users to analyze data and add dimensions easily.
The visual interface is fairly easy to use and the systems appear intuitive to help users in their path to analysis. Illuminate also offers other interesting features as part of additional products offerings:
• iSOLUTIONS—A data management solution for small to medium businesses (SMBs) and business department use.
• iEXPLORER—A user-friendly front end to take advantage of the correlation database without having to work with the metadata layer.
• iLLUMINATE SDK—An application that enables organizations to build and customize analytical applications under the Illuminate platform.
Illuminate’s products run under Intel-based processors, which gives the iSTORE product the ability to be easily installed in Windows-based architectures and to be deployed in a relatively short time.
Is it All about Liberation?
So, what can we gather from the vendors we have featured? A closer inspection suggests they all have something in common: the idea of attempting to liberate the end user from the burden of technical complexities as much as possible. This may not be possible 100% of the time, particularly when end users are looking for a full corporate data warehouse solution, which involves many complex data issues or data quality and data governance issues. But I have no doubt that the data warehouse and BI spaces are currently being reshaped by the interesting products offered by these vendors.
I welcome your thoughts on this interesting topic.
Is it a Database Software like Sybase,Oracle..etc..
Please let me know more about this..
illuminate seems to be similar to what sap is planning to do with HANA (high analytical appliance). Hana is a columnar based data store that can be feed realtime from sap ERP systems. the real speed of analytical capabilities comes from the fact that Hana runs completely in memory (currently up to 2 TB with the most recent blade machines available).
would be interesting to see the differences between Hana and Illuminate
The long term vision of Sap is to put Hana as a DBMS directly under the transactional ERP systems, which in the end means that OLTP and OLAP will run on the same DBMS instance?
Any company has their own specific need in using a effective application software that suits the company’s needs. Data management system requires a massive amount of data entry and requires efficient results that can relay through financial reports. If a application software can deal with that, then the application software developers must have a very good purpose in creating the software.
Markus is correct in the sense that Illuminate expects to be used as an analytical database that runs alongside the OLTP database. Illuminate is not strictly a columnn based store technology. The database is essentially an index and the data values are abstracted and stored as simple values. This allows queries across tables and columns as well as some types of queries that can not be executed in other environments.
Sounds like pie in the sky promises I’ve heard many times over the years, especially from ERP programs that promise the sky, how easy it is for end users, etc etc. The true result is terrible.
I’ll wait for it all to shake out and see what becomes widely used in a few years. Until the the fundamental DW concept will remain in use.
Daves comment on “Pie in the sky” is probably typical of many peoples reactions. In this industry we hear so many things that seem impossible that we become very cynical. However, it is important to remember that sometimes the promises are true. As recently as ten years ago a 10 terabyte database was pie in the sky and today that is middle of the road size.
I am certain that in another ten years, almost all organizations will have an analytical database and none of them will be based on traditional RDBMS technology.