An Imperfect Mirror: How the media and public opinion research industry fail the Canadian electorate.

At the start of the 2008 Federal Election, I was having coffee with a friend who also works in the public opinion research industry. We started talking about what our various colleagues would be doing over the course of the campaign. Given that only a few companies would be doing party work, the rest would be looking to pair up with media outlets. A quick review revealed little change from 2006 – CPAC, the Parliamentary Channel, would continue their relationship with Nik Nanos. CBC would partner with Environics and the Toronto Star would continue with Ekos.

But I was mystified with the approach announced by the Globe and Mail. Although they say they are Canada’s national newspaper, the editors at the Globe decided that they would not commission any national polling. Instead, they decided to focus on “battleground ridings” where, they claimed, the election would be won or lost.

This would be an innovative way at looking at the election contest, but I was disturbed that all the “battleground ridings” selected for this research were located entirely in the suburban areas of Toronto, Montreal and Vancouver. “What about Halifax?” I asked. “With Alexa McDonough gone, all three parties are competitive there. What about Winnipeg or Quebec City? There are definitely seats at play there.”

My friend sighed. “When the Globe talks about “battleground ridings”, they aren’t talking about Liberals versus Conservatives, they are talking about the Globe versus the Toronto Star, the Montreal Gazette and the Vancouver Sun.” In other words, the research that the Globe was commissioning was more about winning new subscribers than about who would be winning seats.

I felt embarrassed about my naïveté. But, on reflection, what other answer was I expecting?

For years now, I have been concerned about the relationship between the major media outlets and the public opinion research industry. It is no secret that media outlets and the public opinion research industry enjoy a symbiotic relationship where the media outlets provide coverage (and credibility) to polling firms in exchange for newsworthy research results, usually in the form of voting intentions or leadership perceptions. Hot polling results helps sell newspapers or attract viewers. It helps a newspaper or television network (or consortium of the same) establish a reputation for having the latest information or, better yet, the keenest insight into the unfolding events of the day.

Being on the front page or in a nationally televised interview promotes a public opinion research firm and its principals in a way that paid advertising cannot. Appearing on the national news or in the headline story gives credibility to the pollsters who, just as one hand washes the other, lend their new-found star power to their media partners.

While this relationship helps both the media and the survey companies reach their corporate goals, there are reasons to be concerned about the way this relationship is influencing our political system.

As with most problems that affect our political culture, this situation did not occur overnight. It evolved out of a promising beginning where the large media outlets would invest in public opinion research to obtain a deeper understanding of the electorate, our issues and our motivations. They would use the expertise of prominent firms to set the methodology, ask questions, analyse data and present the results. The media partner would then determine the newsworthiness of the results and the amount of coverage that would be given to reporting the same.

The first sign of trouble came when the editors and producers wanted to have more control over the methodology that would be used by the research firms, such as the timing of interviews or the questions that would be used. At first blush, this seems harmless enough. After all, shouldn’t the client get what they want?

But we are not talking about a simple client/supplier relationship. We are talking about a partnership where a research firm provides polling results (and their good name) to a media outlet to generate content for their audience. This implies that both partners bring their sensibilities to the table and act in accordance with the tenets of their respective professions.

However, what happens when the desires of the media partner conflict with the standards of their research firm? Usually, it means that the media outlet gets a more compliant public opinion research firm. It is not only the former research firm that loses out; the audience loses the benefit of a more rigorous approach to public opinion.

The relationship between the media and the public opinion research industry has also been negatively affected by the level of resources available to undertake quality research. Even without the recent round of media cutbacks, editors and producers are confronted by limited (and shrinking) resources for their newsrooms. While proprietary polling results often provide the exclusive stories and headlines that most political coverage cannot, there is a cost involved. What is less well-known is that this cost is most often borne by the public opinion research firms seeking profile rather than by the media outlets who want the data.

In the media post-mortem after the 2004 Federal Election, a leading figure in the industry admitted that his firm spent “two or three times” the amount of money provided by their media sponsors for the election survey work that they undertook. In other words, the research company subsidized the research that was “sponsored” by their media partners. 

If there is a general business principle in the public opinion research world, it is the maxim that “you get what you pay for.” While no sensible public opinion firm would cheat its own reputation, the reality is that, with minimal budgets and impending deadlines, the temptation to do only a superficial analysis of the results, often in the form of “horse race” numbers on voting intentions or the rise and fall of political personalities, is strong. The commentary of the researcher that often accompanies the reporting of these results is more often based on speculation than on any hard research results. After all, additional research would add cost and complexity, neither of which is desired by either partner in this process.  

The desire to reduce cost has also driven down the number of people interviewed in surveys on public policy or politics. Fifteen years ago, it was common for public opinion research firms to interview 1,200 or 1,500 Canadians for a national survey that would appear in the media. The current standard for media surveys is 1,000 interviews, which is cheaper to administer.  

This is where the “margin of error”, the basic measure of survey data precision, comes into play. While there may not be a significant difference in the overall margin of error between 1,500 or 1,000 interviews (+/-2.6% and +/-3.2%, respectively), the main impact of these decreased national sample sizes are found in the demographic or regional results that are based on a smaller number of interviews.

For example, British Columbia comprises about 13 percent of the total Canadian population. In a national survey with 1,500 respondents, the BC results would be based on 195 interviews and have a margin of error of +/-7.2 percent. With only 1,000 interviews nation-wide, a total of 130 interviews would be conducted in BC, with resulting margin of error of +/- 8.8 percent.

In either case, the high margin of error in BC means that any change in the results from that province need to be eyed carefully. What may be a significant shift in voting intentions to a reporter or an editor may be nothing more than a change within the margin of error, especially if that margin could be as high as eight points. This detail is often lost on the producer or editor, with the result that survey numbers that should be treated with caution are treated as an absolute fact.

This results in an excess of headlines that are based on nothing more than the higher variation created by results drawn from a small numbers of interviews. An urgent news item that shows a particular leader falling behind in Quebec or a headline declaring a surge in support on the Prairies may be nothing more than a statistical phantom.

Further, with the higher margin of error for these regions or sub-groups, these numbers are bound to change with the next wave of interviews and generate a whole new set of headlines (“Leader rebounds in Quebec”, “Political Chinook evaporates support on Prairies”). Talk about a self-perpetuating media bonanza!

Exclusivity means exactly that – exclusion. Once a media organization purchases research results, even at a deeply discounted rate, they are committed to obtaining the most value they can from their results. They promote their own results and ignore all others in the view that only their survey results provide a true picture of the complex dynamics of the electorate. Of course, promoting your results to the exclusion of all others means that the audience only gets a partial picture of what is going on. This may be good marketing but this practice forms an additional barrier to a better understanding of the electorate.

Taken together, the collusion between the media and their public opinion research partners present a fairly distressing concern for Canadians who want to be informed about their political situation. But there is another question about the ability of the media to serve the information needs of the voting public.

According to the Canadian National Election Studies, which has investigated the opinions of voters in every national election since 1965, arriving at a voting decision is a complex process for many voters. Many factors are taken into consideration but, overall, the policies promoted by the parties and perceptions of their ability to deliver on these policies account for about one-half of a voting decision for an individual voter. The remaining one-half of a voting decision is almost evenly split between perceptions of the party leader and the local candidate, with the local candidate a more important factor in rural ridings and leaders more important in multi-seat urban areas.

It is a reality that the media, especially television, has a difficult time covering or explaining the policies promoted or defended by the various parties. There is the difficulty of trying to compress complex policy nuances into a few paragraphs of script or column inches. Even in the so-called “New Media”, the policy differences between the parties rarely receive coverage, let alone the critical coverage needed by the Canadian who wants to cast an informed vote.

Stories that focus on personalities and interpersonal conflict have more sizzle than substantive features that contrast and compare different party platforms. While party leaders account for about one-quarter of a voting decision, media analysis from past elections shows that stories about party leaders and other personalities dominate media coverage. Even in the “Free Trade Election” of 1988, coverage of the merits of the various platforms proposed by the parties took a distant back seat to the coverage of the party leaders.

One television reporter told me of a talk given by a senior network editor to the local news team on how political reporting would change within their organization. This new way would emphasize “politics without process” and “conflict without context” to make the stories sharper and less dependant on the knowledge of the viewer. The focus would be on the here and now, rather than on how these current events reflect the past or how they would affect the future. In other words, completely disconnected from the continuum of existence. This is an apt parable for the way that the media and the public opinion industry are failing us as voters.