BEWARE THE NUMBERS
Marketing Magazine May 17, 2010

Not all research studies are created equally, say CTV’s ROB DILWORTH, Canwest’s KATHY GARDNER and TVB’s THERESA TREUTLER

Marketers and advertising professionals are often faced with research studies that differ widely from, if not conflict with one another.

This should not come as a surprise since “all [research] methods are fundamentally compromises of reality,” according to Harvard professor Jerry Zaltman, speaking at the recent Advertising Research Foundation conference. Making all of your decisions based on just one research study is very risky. Zaltman and his panel of neuroscientists instead recommend the “convergent validity” approach as the only way to get to the truth. In other words, advertising professionals should take the time to look at all relevant and accredited data sources before important decisions are made.

For years, studies have been published that infer the demise of television due to increased usage of the Internet. While these studies might make headlines, insufficient attention is paid to their context. For example, one study was actually an online survey of online users (using an opt-in panel). Using a survey like this to ask how much time people spend with online versus offline media will produce very different results than a rigorous survey balanced to the population, including those who have no home Internet connection.

Unfortunately, most of the research we hear about is based on consumer surveys with small sample sizes. It is a simple fact that good media research is expensive. Media planners rely on side-by-side comparisons of media usage data with much larger sample sizes to guide their thinking. These types of comparisons are provided within RTS, PMB, and NADbank databases.

While the sample sizes for these databases are much larger than the studies we typically come across in the daily press, RTS, PMB and NADbank are all based on self-reported surveys. Self-reported surveys result in people reporting what they think they do, rather than what they actually do. This is because remembering how much time you spend with the various media is difficult. In some cases this leads to people over reporting how much time they spend with some media and under reporting the time they spend with others. In a side-by-side comparison, we analyzed the “time spent” data available within each database. Below are the results for some key demographics:

Table 1- Average Weekly Hours (see notes)

It is apparent that each research study produced different results, and in many cases the differences are vast. For example, PMB reports that adults 18+ use the Internet an average of eight hours a week, while NADbank data shows they use it 22.9 hours, a difference of 14.9 hours.

The broadcast industry is now using measurement technology that is intended to address the challenges of self-report methodology. Radio and TV broadcasters now rely on Portable People Meters (PPMs) to measure what people actually do. The new system is mostly passive; panel members just need to wear the pager-sized device throughout the day and return it to its docking station when they go to bed. The PPM unit “listens” for any encoded TV signal that is audible to the wearer, including out-of-home TV viewing at work, school, bars and restaurants.

Since PPMs do not rely on human input, personal bias is minimized. PPMs thus exploded some of the myths about TV as it is all about measuring real people, watching real television, in real time. PPM data shows that Canadians 25-54 watch an average of 25.7 hours of television weekly, 6.7 hours more than reported by PMB (19 hours). In addition, the coveted 18-24 demographic watches an average of 25.4 hours of television weekly, according to PPM, 6.4 hours more than reported by PMB.

The bottom line is that not all research studies are created equal, and research must continue to evolve to capture the true media habits of Canadians. And while PPM is a big step forward, we have more work to do. For now the truth lies within the “convergent reality” of all accredited studies that decision-makers need to consider relative to their communication objectives and media decisions. Simply put, your common sense and good judgement are irreplaceable when evaluating research study learning.

Rob Dilworth, is Vice-President Market Research, CTV Television Inc.; Kathy Gardner, is SVP Strategic Insight – Research, Canwest Sales & Marketing; and Theresa Treutler, is President & CEO, Television Bureau of Canada.