Research for one of my projects led me to ask both software vendors and customers about the factors most important to software users in the selection of a business intelligence (BI) solution. Two topics resounded: the use of BI tools to improve data management and business performance management. Consumers are continuously looking for innovative ways to move, store, and improve the quality of their data as well as to ways to capture the most valuable information for improved decision making and business performance.

Let’s look at what the data tells us. Below is a Pareto chart based on data from Technology Evaluation Centers (TEC) on what software users consider to be the most popular functionality features for BI applications. For those of you who don’t know, the definition of a Pareto chart according to Whatis.com is “a vertical bar graph in which values are plotted in decreasing order of relative frequency from left to right. Pareto charts are extremely useful for analyzing what problems need attention first because the taller bars on the chart, which represent frequency, clearly illustrate which variables have the greatest cumulative effect on a given system.”

From the chart we can see that data management and business performance management (BPM) together make up nearly 60% of all responses. This means that these two topics are still of major concern for most organizations and represent the top priorities for companies in search for a BI solution. What are the root causes for this unanimous perspective?

Why is data management still an issue?
Turning data into valuable information has never been more important, as organizations today must be extremely cautious when managing information—its movement, analysis, presentation, and security—owing to local as well as global legal and economic considerations (new law regulations, new business models, etc.). Technologies and frameworks like the data mart and the data warehouse are addressing some of the back-end information management issues that support the decision-making process. Possible root causes for the quasi-perpetual data management issue relate to the natural evolution of data within the organization:

1. Diversity. Data is being generated from different channels and its nature has changed. In the past, data originated mostly from traditional relational database systems (RDBMSs), but now it also comes from popular social media channels, and thus can take various different forms. Though this information has implicit value, oftentimes it cannot be processed with existing data management applications.
2. Volume. The increase in the number of data sources and the globalization and diversification of businesses have led to the exponential growth of data—resulting in massive amounts of data to be managed.
3. Velocity. As data volume increases, so must the speed at which data is captured and transformed into its final form. This is crucial for time-sensitive businesses where operations and analysis come together (e.g., retailing and support center systems as well as systems based on real-time event analysis in the oil and gas industry or in manufacturing).
4. Sophistication. With the increasing complexity of data, high data quality and security are required to enable data collection, transformation, and analysis to achieve expedient decision making.
5. Applicability. These aforementioned factors can compromise the applicability of the data to business process and performance improvements. Some organizations find it difficult to closely align data management and business cooperation, tactics, and strategy.

In addition, appropriate and expedient data management must involve the necessary technology (hardware and software) and the right people and business processes working together for the sole purpose of enhancing business performance. This is simple to say but hard to do, for as James Robertson states in his blog post “10 principles of effective information management”:

 Information management issues can be overwhelming

And

There are no simple answers to complex issues and needs

Today, managing data is more challenging than ever before owing to

a) the rapid pace at which business rules and its technology components change

b) the advent of social media channels, and

c) the development of new technologies involving structured and unstructured data.

As a result, organizations have to comply with increasingly complex business requirements, which are harder to fulfill and require faster time responses.

One option: divide and conquer
Perhaps we should start thinking about data management more in terms of an initiative than as an isolated product or technology. The root causes of data management issues we have identified suggest that these issues will persist owing to the nature of the data itself.

So, before even considering changing the existing data management infrastructure, it might be worthwhile to go through the following checklist:

1. Prioritize data management issues to be addressed.
2. Focus on first and second priorities, according to the Pareto graph. This will ensure you’ll cover the most outstanding issues using an approach based on system evolution and or scalability, such as an agile or incremental project development framework.
3. Decide upon a strategy to manage complexity and mitigate impending issues.
4. Plan to address priorities according to business needs and improved customer experience, thereby improving likelihood of user adoption of the new system. The ultimate measure of the success for any software solution is its wide adoption by users.
5. Assess the effectiveness of each step in the data management process, promoting learning and process evolution—a data management solution is not static, but it grows and evolves to meet the constantly changing conditions and requirements.

A data management process is not a mere information technology (IT) project initiative. A joint and collaborative effort from both the business and IT segments of an organization is needed to achieve a successful data management initiative.

Conclusion
So are we any closer to addressing all the data management issues impacting the performance of organizations today? Probably, but there is no single approach that will address them all. The most important thing we can do to ensure a more efficient data management process is stay attuned to the changing business priorities of the organization.

As always, I welcome your thoughts—leave a comment below, and I’ll respond as soon as I can.