Advanced analytics is the next big thing. If you have not budgeted to add predictive and prescriptive analytics into your analytics arsenal, you are apparently already lagging behind your competitors according to a recent Gartner Predicts 2016: Analytics Strategy paper. Where traditional business intelligence, self-service reporting and data discovery tools like Tableau, Qlik and Power BI provide enormous value in the ability to understand and summarize what happened in the past. Predictive and prescriptive analytics are game changers that can intelligently optimize future decisions.
Research suggests the global predictive analytics segment will increase from $2.74 billion in 2015 to $9.20 billion by 2020 at a compound annual growth rate (CAGR) of 27.4%. The cognitive analytics space is estimated by IDC to become a $60 billion market by 2025.
Exceeding current capabilities
Analytics has become a competitive differentiator in a big data world that is quickly transforming into the digital business era. The vast amounts, complexity and variety of data sources in our data-driven economy are already overwhelming existing self-service BI/data discovery tools that can’t handle or render the data. The human analyst or decision maker’s ability to manually identify new insights or detect changing patterns efficiently with those tools when presented with hundreds of variables is also being exceeded. Most decisions can’t wait for scarce data scientist talent to evaluate potential options. As a result, the business ends up operating on riskier gut feel rather than data-driven decisions.
To address these challenges, organizations are embracing aspects of analytics automation. They are also using predictive and prescriptive APIs to embed sophisticated, real-time intelligence into business user reports and applications. Instead of business users digging into historical data or navigating to a specialist data science tool, unbiased prescriptive guidance is being delivered at the time of decision…where the business user is making that decision.
For example, predictive lead scoring is being embedded within CRM systems to objectively rank marketing qualified incoming leads to allow sales teams to focus their time on prospects that are most likely to buy.
Why adopt now?
Even if you are struggling with historical self-service reporting and dashboards, you can still enjoy the transformational decision-making impacts that predictive analytics delivers. Truth be told – you will always be iterating on historical reporting. Predictive analytics is likely to be a far more strategic investment of your limited time and resources. Automation and predictive pattern detection can eliminate weeks of manual effort that still might not locate statistically relevant insights.
The analytics competency conversation has risen all the way to board and executive levels. Today we are seeing new C-level, Chief Analytics Officer (CAO) and Chief Data Officer (CDO) roles created to maximize the value of organizational data assets, improve decision making and enhance business processes. In the industry best seller, “Competing on Analytics: The New Science of Winning”, the authors do cite high performing business processes being among the last remaining points of competitive differentiation.
Aside from the data explosion dilemma, predictive and prescriptive analytics are becoming critical success factors for long-term survival. No one wants to become the next extinct business, solution or service that missed the early market signals and opportunities to evolve before it was too late. By relying only on historical reports, gut feel and intuition instead of statistical pattern detection, probability distributions and what-if scenarios, risks and opportunities are likely to be underestimated.
According to TDWI Research, the top five reasons why companies want to use predictive analytics are as follows.
- Predict trends
- Understand customers
- Improve business performance
- Drive strategic decision-making
- Predict behavior
Market shift towards smart data discovery
Another market force that is driving the momentum of advanced analytics adoption is the introduction of an impressive new generation of smart data discovery tools including IBM Watson Analytics and BeyondCore. In the past, the learning curve to get any value out of predictive tools was far too steep for most business users. Today that is changing.
Smart data discovery tools deliver user-friendly advanced analytics functionality that is designed specifically for the non-data scientist, information worker. Unlike the current self-service BI/data discovery tools that include limited forecasting, trend lines, outlier highlighting and other basic insight detection, smart data discovery tools provide intelligent data preparation and much deeper, statistically significant, guided diagnostic exploration. Another key difference is that business users can experiment with what-if scenarios and get unbiased, detailed written interpretations of prescriptive actionable insights.
For those of us that have gone through the traditional IT-led BI to self-service BI/data discovery revolution, the smart data discovery market shift towards business users from the data science professional may sound and feel familiar. I am hearing similar fears, objections and adoption dynamics that I heard in the past. Just like self-service reporting, citizen data science will require training and governance for successful enterprise-wide deployments.
Don’t overlook data preparation differences
As organizations venture beyond data visualization to pilot the power of smart data discovery and predictive analytics, existing skills such as data querying, cleansing and transformation (ETL) will easily transfer. Other skills like the art and science of data preparation for predictive modeling will require additional training and resources to achieve the best results. Don’t assume that you can merely connect a smart data discovery tool to a spreadsheet, dimensional data warehouse, OLAP cube or OLTP transactional database, click an easy button and get fabulous findings. The beauty of the human mind for business process subject matter expertise is still needed to get the most value from these tools.
Predictive data preparation does require a different thought process to model the business question than historical reporting and dimensional data warehousing processes. The predictive data preparation approach is similar to data preparation steps in statistical analysis. Flattened single tables, views or data exports that contains an array of derived columns called features to illustrate a decision process are created. You don’t want too many features in your input data set or your model will suffer from overfitting issues. You also don’t want too few features since that would not be useful. Thus for all the advances in analytics automation, predictive data preparation is still truly an art and a science.
In a future article I will cover predictive data preparation tips and examples to jump start successful evaluation of smart data discovery tools. If you want to explore this topic sooner, check out Dean Abbott’s latest book called “Applied Predictive Analytics: Principles and Techniques for the Professional Data Analyst”. Dean dedicated an entire chapter to predictive model data preparation that can help you make the leap from descriptive to predictive data preparation.