Aug
17
Data Discovery Applications
Filed Under (BI and Performance Management) by Jorge Garcia (see bio)
 

Over the last five years, the business intelligence (BI) space has undergone a huge transformation. The business user community has lobbied for data analysis tools that are easier to use, agile to deploy, and less expensive, thus encouraging the emergence of new products and vendors. These conditions, along with the acquisitions and mergers of software companies looking to offer new applications for analysis and decision-making support, are breeding a new set of innovative tools: the so-called data discovery applications.

What are they?

Data discovery tools are specifically aimed at connecting users to a wide variety of data source types (structured, semi-structured, and non-structured) and enabling users to freely explore the data within. There are no predefined data drill paths, so users can interact with data the way they want to and easily create visualizations that suit their own purposes. As such, they boast a flexibility and freshness that traditional BI solutions might find hard to match.

Taking advantage of these features, users can create prototypes fast and evolve them into more robust data exploration and visualization projects if needed. This way, costs and efforts can be kept to a minimum, so organizations of all sizes can benefit from data discovery tools, whether using them as a basis for creating more robust BI applications or maintaining them as simple solutions in themselves at low cost.

In the past, data discovery tools were limited in functionality and tied to strong data visualization features. They have at times been misunderstood—the true potential of data discovery applications has been neglected, and they are often considered a “lite” version of more serious BI applications. But certain innovative software providers such as Lyzasoft, QlikTech, and Endeca (now part of Oracle), and the research they have produced, support the principle of giving non-technical users the freedom to search/locate and explore information in a friendly but serious way to uncover insight.

Over time, as more and more organizations saw the potential of these tools to deliver quick and versatile data analysis services, data discovery applications were able to offer new features and functions that gave them entrance into the mainstream of the BI space.

Some of the main differences between traditional BI solutions and data discovery tools are as follows:

  • Traditional BI solutions consist of a complete stack of features covering a complete cycle—from specialized data movement tools (data integration, data quality, etc.) to data analysis and visualization. Data discovery tools use best-of-breed technologies to cover mainly analysis, exploration (search, drill through, etc.), and visualization.
  • Data discovery tools enable discovery “on the go,” letting users explore data and discover insight without having formed a specific question. BI applications tend to follow a stricter methodology, with structured paths for exploring the data and following well-defined rules.
  • Deployment of data discovery tools tends to be faster than for traditional BI tools, which require more preparation to deploy and operate.
  • Data discovery tools are extremely user-oriented, while traditional BI tools take a corporate approach and so are somewhat more challenging technically to learn and apply.

With the concepts of ease of use, freedom to explore, and self-service at their basis, these business applications contain features that enable organizations to quickly gain access to a broad set of data sources, and start exploring data, analyzing it, and gaining insights from that information. Some of the highlights among the functionalities that data discovery applications offer include the following:

  • Easy-to-deploy data integration capabilities
  • Exploration capabilities of both structured and unstructured data
  • Rich drill-through capabilities
  • Easy and powerful visual data modeling and visualization tools
  • Features for enabling data mash-ups
  • Rich enterprise data and content search capabilities
  • Text enrichment and analysis features such as sentiment analysis
  • Team collaboration features
  • Availability of rich analysis tools such as regression, nonlinear modeling, etc.
  • Easy integration with Microsoft Office, especially Excel
  • Mobile analytics features

Organizations can benefit from these types of applications in the following ways:

  • Rapid deployment at a lower price than for corporate BI suites
  • Easier application of incremental and agile data analysis applications, such as prototyping and agile methodologies
  • Reduced need for supplementary code, or even eliminated altogether

Some interesting aspects of data discovery tools, if used properly, include their ability to uncover data relations and easily integrate dispersed corporate data, enabling a corporate snapshot and rapid analysis of information, potentially speeding up the time-to-decision cycle. (But it is important that these applications be deployed with the same rigor as any other software application within the organization to ensure that data is at all times safe from internal and external damage, and that it complies with regulations if necessary.)

Here are some currently hot data discovery products:

Many software providers of data discovery offerings provide both server and desktop options. Others, such as IBM, Oracle, and SAP, provide strong connection and integration of their data discovery tools with their existing BI platforms, giving customers both the flexibility to work on their desktop and the power of heavy-duty BI solutions if needed.

A final word

Nowadays it is hard to classify business analytics products as they touch upon so many different sets of functionality. Such is the case with QlikView, Tableau, and Endeca, which fulfill functions that might classify them in other areas of the business analytics space. Many vendors offer data discovery applications alongside their traditional BI suites.

Will data discovery applications replace corporate BI platforms? Maybe. For now, some organizations (especially large ones) rely on full-fledged traditional BI applications to cover all their BI needs, but data discovery tools are gaining presence in the small to medium business (SMB) market. They are often used as a solution by lines of business and can be an extremely useful complement to traditional platforms used in large corporations. (We’ll have more on data discovery tools in our buyer’s guide for BI and data management, coming this October.)

Data discovery applications can bring many benefits to an organization, enabling dynamic and versatile analysis of data, but to successfully meet the data analysis needs of your organization those applications need to be considered with the same rigor as you would employ in acquiring and implementing any other type of enterprise software.

What do you think? Are you using one of these products or some other data discovery tool? Are you a software provider with an offer we should now about? Please let us know.

Share This

Comments

Adrian Parker on 20 August, 2012 at 5:35 am #

Data discovery is only the beginning and for the most part is akin to a spreadsheet on steroids. By enabling decisions to be made sooner and more accurately tactical use of such tools can add huge business benefit.

The true strategic value of such tools is to extract actionable insight or as QlikTech call it Business Discovery. Business Discovery enables change of business behaviour to take into account both internal and external factors affecting company productivity and performance.

We see data discovery and business discovery tools as being the must-have tools given the explosion of corporate data, as we move into a world of fragmented data stored in multiple data sources.

These technologies enable too; We can now measure for example sentiment analysis via interpretation of on-line comments and feedback in a real time way. Look at Geospatial data analysis - presenting visually the trends we see in numbers.

Exciting times indeed for modern agile analysis tools being more “nimble” than “lite” in my opinion. There is something for everyone meaning that the topic of Actionable Insight (Business Discovery) should be on the strategic agenda for every company.


Brent Hills on 20 August, 2012 at 9:19 am #

It continues to amaze me (in the 25 years that I have been involved with BI and Performance Management) how most focus in the IT community is on the analytical tools rather than the users education. I have seen this focus first with numerical and textural analytics in the 90’s and now I am seeing it again with video analytics. Surely will will learn from our past errors.


Jason Beres on 25 September, 2012 at 11:20 pm #

This article nails it. At Infragistics (http://www.infragistics.com), we’ve built developer tools for years that help in the creation of tools like you’ve highlighted, but more so, are used to give the developer the job of creating dashboards, reports, etc, based on capabilities of the tools. This fits into the restrictive corporate BI model you’ve highlighted. At the end of the day, no matter the effort of the development teams to deliver sophisticated tools to build dashboards and reports, the business user will never be satisfied to live w/in the limitations of the report. Ultimately, they want to explore the data themselves, freely, on their time at their pace to understand their data and gain the insight they need.

The future is in tools that give the business user friction-free access to data, and an intuitive user experience to understand the data, make better decisions, and then share those insights with their teams in an easy way. This will always include rich visualizations & drill-downs, the goal is to be able to quickly understand complex data, and nothing does that better than a well-designed visualization.

We’ve just shipped a tool called ReportPlus, which allows anyone to explore and understand their data and build their own dashboards all on an iPad. It seems almost every executive has an iPad nowadays, and managers have them, and enterprises are buying them like they are going out of style, it’s a natural move to use the iPad for more than just corporate email. Our SharePlus product, which is a native SharePoint experience for the iPad is pushing 700K users, which shows they popularity of iPads in a corporate environment (SharePoint = Corporate). The more business users see the value of tablets to do real business functions; they will demand the sort of data discovery tools and concepts you’ve talked about in this article.

I hope this doesn’t sound like an advertisement; this is topic we are very passionate about and believe in strongly. We strive that our tools will achieve what you’ve so nicely laid out in this article. Thanks for the great article!


Richard Blakemore on 26 December, 2012 at 6:10 am #

Let me declare up front that I am a consultant that implements QlikView solutions. I came to QlikView from a fairly serious love affair with MS Excel, and discovered something that was easy to use, and so much better and bigger than just a spreadsheet on steroids.

I find it interesting how many of the stack vendors like to characterise the data discovery tools as “lite” or “just a visualisation layer”, or the ultimate putdown “does not scale well”. I also find it interesting how the stack vendors have rushed to commandeer the language and concepts that QlikView played a massive role in developing and has used so successfully since the 1990s.

Technologies like “column/vector based databases”, and “in-memory processing” are now becoming all the rage with SAP rushing HANA to market, and the other stack vendors pushing their pre-aggregated cubes up into memory.

The net effect of all of this is a serious muddying of the waters - perhaps deliberately by the stack vendors. Columnar databases beat row based databases in data extraction speed tests by factors of 10 or more. Over millions and billions of rows, this is a massive bonus. Pushing a pre-aggregated cube up into memory still means that the user is bound by the rules of the pre-aggregation. So if they want something else, they have to go back to IT to get it. (I find it interesting that Microsoft does not have a columnar database capability (other than powerpivot, and the ability to add a single columnar index to a table - which makes the table read only)).

Those going for a traditional stack vendor solution with a row based datawarehouse, and cubes smashed up into memory are finding that their reporting/BI solutions are taking 6-24 months to build, and then they run like a wet week over a fairly narrow set of parameters. In the meantime, teams of business analysts are building Excel Hell to try to reduce the top floor screams for reporting down to a jet engine whine.

Those going with QlikView are getting solutions that allow drill down to transaction level from any angle within the data in days and weeks. This almost silences the screaming for reporting from the top floor, lets the people who know the data actually explore it on their ipads, desktops etc.(without having to go back to IT to build another cube), and creates space so that data warehouse projects can be properly spec’d and built - without pressure from upstairs to make it happen quickly because all of the reporting is waiting on it.

In my humble opinion, data or business discovery is what BI should always have been, and the combination of the powerful ETL engine, and data visualisation layer that is QlikView delivers results to end users significantly faster than any other tool I have seen.


4265. on 20 March, 2013 at 5:30 am #

4265….

Data Discovery Applications » The TEC Blog…


hurtig lån on 10 April, 2013 at 9:33 am #

hurtig lån…

Data Discovery Applications » The TEC Blog…


*Name:
*E-mail (private):
Web site:
*Comments: