Are you following “Reliability” off a cliff?

Observation: Emphasis on reliability over validity unwittingly stifles Marketing innovation.

Roger Martin (former Director of Monitor Company and Dean of the Rotman School of Management) teaches the following lesson on reliability and validity:

  • Reliability focuses on producing a predictable, consistent outcome. The IQ test focused on ‘little logical puzzles’ is highly reliable. “The problem is it doesn’t really predict much about how you’ll do in life – other than that, it’s really good”.
  • Validity focuses on producing outcomes you really want. “Emotional Intelligence is a less reliable test… you tradeoff some reliability to get something that takes more things into consideration – including fuzzier variables – to get something more valid”.

Roger asks: “Do you want consistency or do you want an outcome you’ll really like?”.

In the world of Marketing, especially in large companies, I often see Reliability trumping Validity. One example of this is following marketing mix models (MMM) ‘off cliffs’. MMM’s are, like IQ tests, one great tool to aid judgment. The tool has many strengths and important limitations. Much like an IQ test reveals nothing about a person’s people skills, MMM reveals nothing about how a highly targeted campaign aimed at new household acquisition performs over an appropriate time period. Yet many companies have made MMM the one holy grail upon which all spend decisions are based. This feels safe (nobody ever got fired for MMM based spends), but leads to bad decisions. Rather, companies need a suite of analytical tools to aid judgment in making valid spend decisions… decisions that produce the desired outcome.

Another example is in the targeting realm. Using a quant segmentation study and picking 1-2 of 5 segments is very simple and reliable, and has some predictability. We know a lot of facts about those people and can measure how we’re doing against them. However, to make valid target choice (one that delivers the desired outcome), looking at consumers through different lenses, including some fuzzy ones, is necessary.

The other danger of Reliability you might want to address is that reliable systems assume the conditions of the past are unchanged in the present and future. MMM based decisions are ones based on history in a world that is changing more rapidly than ever.

Interestingly, Marketing leaders prioritize validity in recruiting. A suite of tools are leveraged including: on-line screenings for multiple intelligences, interviews, formal reference checks, informal reference checks, etc… Why do we accept a Valid, albeit less reliable, system in this arena, but follow Reliability off a cliff in multi-million dollar spend decisions… with far greater financial risk than a bad hire?

Roger notes that “Over time, forces of Reliability tend to overwhelm forces of Validity in large companies… and it often takes the CEO to force Validity back in”. How does this play out in your company? What might you do to “force validity back into the system”?