Big data will change the way that business is done forever. It will revolutionize marketing, product management, shipping, pricing, and even customer service. The sheer amount of data coming in will provide companies with more predictability and customization power than ever before.
Where did big data start? Big data started with the internet and the tracking of people on the internet through web analytics. Web analysts were the trailblazers of big data and the first to start to explore how to use the power of data to predicting people’s behavior, desires and needs.
I’ve started to notice in my career a similar progression for any business decision as it slowly flows from a non-analytical business owner making ‘gut calls’ to completely data-driven decision making. This transformation is widespread, occurring everywhere from marketing investment decisions to HR recruitment analytics to predicting IT system failures. For the purposes of simplicity, I’m going to talk about marketing campaign decisions as an example — but the process seems to hold true no matter the context.
Think of the data available to marketing analytics back in the 1960s. If you’ve watched Mad Men, you know how much gut instinct went into decision making. In one episode, an analyst gets dismissed by Don Draper as knowing very little and being of almost no help. Those gurus, who may not have even been able to articulate why they were making decisions, were the trusted leaders. But, suddenly, relevant data becomes available and people begin to dream what this data could provide.
The first step is reporting on trends. As the trend reports come out, the business starts to see connections between some trends and business performance. This then leads to Key Performance Indicator reports where a metric or set of metrics has gained business acceptance. At this stage in the analytics evolution of a business, a report that referrals are down this quarter might cause concern that sales will miss their target. Why are they concerned? Because there is a belief that the metric is connected with performance. This is an important and necessary step to move into the next phase.
Business Intelligence or Data Exploration is where you start to really dive into the data and combine it with the knowhow of those who have been doing the work for 20 years — the experts of the company who have an instinctive feel for why one campaign will work and another will fail. This stage can also be critical to helping the analyst understand and dissect some of the decision making and for the current experts to start trusting the analyst team to understand of the intricacies of the business. Sometimes this phase is skipped… but in my experience, this can be extremely detrimental to the progression of analytic decision making in a company, resulting in a more adversarial relationship, with analytics fighting traditional business decision makers and the experts they’ve relied on in the past.
The most exciting phase for most analysts is once the business is ready to not just use data to figure out what’s going on now, but to also use it to ask ‘what if?’: predictive modeling. This is the phase where most statistics majors get excited – where advanced modeling can be created to predict which campaign will perform best, which IT systems are likely to fail, which patient is most likely to develop diabetes in the next year. As these models continue to be developed and prove their worth, the next phase comes into play: the data is collected real-time and decisions can be made automatically in response with analysts only monitoring the system. New data continues to come in, and results in an iterative approach. As new or more detailed data sources come in, we still go through the phases (Finding Trends, KPIs, Data Extraction, Modeling), but the integration of the new data goes faster.
At each point in these phases, the acceptance of the analytics from the business is an important part of progression to the next level. This has become easier as other companies and industries have gained public renown for their use of analytics, and provided pressure to follow suit… but still there can be resistance.
Ownership of the business decision making also follows a phased progression eventually ending with the analytics team taking over ownership of the business decision. Let’s take a brief walk down history lane. Pricing analytics actually first started in the airline industry. The idea of pricing based on plane fill rate and timing came up and some executive was willing to give it a test — the first phase of the movement towards analytics under ownership of the decision. Assuming the test is successful — as it most definitely was in the airline business — the business decision maker is willing to pass along more control and suggests developing across all markets or lines of businesses, but isn’t ready to hand over the reins, and wants to stay in the loop as a decision approver. Eventually, even this approval effort becomes a formality as the analytics increases in sophistication and the third phase is reached where the analyst has the approval and is expected to review all decisions made by the model. Eventually the speed and quantity of incoming data becomes great enough, and model sophistication becomes strong enough, that no human intervention is required in the business decision, only monitoring to improve the models.
As an exception that proves the rule, here’s an example of where algorithms designed on models to set pricing resulted in an out of print book soaring to $2 M dollars before the analysts noted it and modified their pricing model. Quite fun! It shows the exciting and somewhat silly decisions that we have to look forward to as more of our day to day decisions become machine and model driven.
I think it also shows the importance of having analysis define reasonable upper and lower bounds for the decisions that come out of their model — it’s the lack of these that cause things like flash crashes in the stock market.
Eventually there will be complete integration of the business decision making and the analytics. We are already starting to see this in several areas including pricing and digital marketing. Can you imagine someone being in digital marketing that doesn’t understand page views and referrals and feels comfortable looking at numbers and predicting future impacts? I cannot. This shows how integrated analytics has become with business decision making in the digital marketing space. I believe in the next 5 years, this blurring of analytics and business decision making will continue to take over other areas as well. It’s a great time to be in analytics!