Many people look at work done in the past, find flaws, and are quick to throw away the old in favor of the new. I’m all for evolving our thinking, but I find it difficult to stomach when an analyst or analytics consumer says flat-out, “This measurement is pure BS and doesn’t work at all.” In most cases, there’s a reason why a given measurement received enough acceptance for business to depend on it for decision-making. The measurement may be outdated, and there may be ways to evolve the approach — but it’s important to understand the history before you toss the old measurement away.
Here’s the life of an analyst: You have TONS of data. An avalanche of new measurements. Your first approach is to sort it, and segment the data into categories so you can start to see performance differences in the segments. As your understanding of the data evolves, your measurements also evolve. This is the approach that Adolphe Quetelet took when he created the BMI (Body Mass Index) metric in the 19th century. He had a fair amount of data on height and weight and created a metric that correlated with diseases like type II diabetes. The BMI was proven to be such a useful and predictive indicator that it was factored into life insurance policies post WWII. In the mid-1990s, it became part of the World Health Organization’s approved metrics for obesity, and even popular with members of the general public as their doctors discussed their BMIs with them.
A large part of why BMI was so successful was the ease of measurement. Height and weight became data points easily captured at a doctors visit and were strictly objective measures.
BMI is still widely utilized for trend-setting, population demographic measurements, and in predictive models to determine disease propensity or progression. It performs very well in all of these and will continue to be a worthy measurement for many purposes.
This same principle could hold true with online marketing analytics and click-through rate (CTR). CTR has often been the golden metric for online advertising – but while CTR is a good metric for seeing trends and getting an elementary level of understanding of the data, our application of the metrics must change as our understanding and use of the data does.
The pushback on BMI came when it was considered at an individual level. For statistical samples, BMI can be a great predictor of how many people in the sample will get a disease… but on an individual level, there are so many things that can impact the BMI metric that it might very well be useless for a particular individual.
For example, one person could be an outlier to the model. The general assumption is that BMI is a measurement for obesity, but by purely considering height and weight, it fails in unexpected ways, and for some of the healthiest people, like bodybuilders. Dwayne “The Rock” Johnson has a BMI of 34.3 – ‘obese’ by the BMI model!
Be the hardest (and smartest) worker in the room. #AndHaveBigBalls #TrioForSuccess #SillyWorkoutFacesAreOptional
Should The Rock’s doctor tell him he needs to lose weight? In the same way that you wouldn’t want to hold ALL patients accountable for their BMI in the exact same way, you must not blindly treat all CTRs equally, or you might find out too late that your advertising campaign had the CTR equivalent of bodybuilders – great health, just with different goals.
If, for example, you are running an online campaign with the purpose of branding (where CTR is not as desired as impressions), you may have a stellar campaign with a weak CTR. Should you cut it? “Yes” is the easy answer… but what if I told you sales numbers have increased significantly since the campaign started?
Does this mean CTR is a bad measurement? No, not at all. It is simply one measurement of many, and a very strong one that the industry gravitated to early because of the ease of capture and strong correlation to great campaigns. It is not, however, the only measurement and simply having a low CTR does not necessarily mean that campaigns need to be revamped.
This content was created per request of Multiview and included in their blog posting. Please see the resulting interview and original post here.