ANALYSIS: Should we trust all of these polls?

On Oct. 17, yet another poll was released to the media. It showed that mayoral candidate John Tory is squarely in the lead, six or nine percentage points up on his competitors. However, it all depends on your source.

Since the candidates started campaigning in January, Toronto residents have been inundated with polls by Forum Research Inc., Nanos Research and others, that claim authority on an ultimately unpredictable future.

So what makes a poll valid? Can we trust pollster data? When it comes down to it, how much stock can we put in polls? As Doug Ford says, “The only poll that counts is on election day.”

The Ryersonian answers your questions about polls.

Is sampling random?

There are a number of reports on social media from residents saying they have been contacted multiple times to participate in polls. One user called Shizzla, posted on the Toronto Reddit page that he or she received two identical robotic calls asking the same questions. This led the poster to ask: “How accurate are polls again?!?”

“The short answer is—it depends,” said Andrea Noack, a Ryerson sociology professor who specializes in quantitative data analysis.

According to Noack, a good poll uses a big sample and publishes the margin of error. The problem is that the non-response rate is so high, that polling agencies have to cast a wide net in order to meet their sample size requirements.

In order to reach a sample size of 1000, a polling agency must contact anywhere from 30 to 80 thousand people, sometimes more.

“People have turned to other methods because it’s increasingly hard and incredibly expensive to do the randomized polls,” explained Noack.

Éric Grenier, the founder of polling site,saidthe response rate can be as low as one or two per cent.

As for Shizzla’s complaint, Grenier said that the sheer volume of calls necessary to conduct a poll means that some people may be called more than once.

“The odds of being called again in a smaller population is much higher,” he said. “But it still shouldn’t happen.”

Noack adds that with three to five different agencies conducting polls in Toronto, there is bound to be overlap in their randomly selected sample.

Who is getting called?

People with landlines, and who are likely to be home during the day, get called the most said Noack.

He said landline and cellphone users are distinct demographics. Cell users, she said, “Tend to have lower income, they tend to be renters, they move more often and they are likely to have different political preferences than those with landlines, who tend to be older.”

“As far as I know, there is no sampling frame for cellphone-only samplings in Canada, partly because the cellphone companies don’t release their numbers,” said Noack.

Both Forum Research and Mainstreet Technologies said they contact everybody, including cellphone users.

What makes a poll accurate?

There is debate among polling agencies about how important the sample size really is.

“It’s a mathematical certainty, not an issue of opinion,” said Lorne Bozinoff, president and founder of Forum Research. “Someone can have a larger sample size and still be less accurate.”

The sample size determines the margin of error, something that David Valentin, spokesperson for Mainstreet Technologies, said is one of the most important factors in poll accuracy. The margin of error calculates the interval of the true percentage, to compensate for weighting discrepancy. The larger the margin of error, the less exact the numbers will be.

In a poll where one candidate sits at 40 per cent and another at 30 per cent, a margin of error of plus-or-minus four per cent can mean a 10-point lead or a two-point lead. The exact numbers could be 36 per cent and 34 per cent for these two candidates.

“You want to decrease (the margin of error) as much as possible. It is better to be actually correct than mathematically correct.”

The most costly part of a poll is labour, which adds up when a polling agency insists on a large sample size.

The alternative is to use a system called interactive voice response, an automated system which allows residents to choose their response from a series of predetermined values.

“The issue with those is not about the sampling, but that anybody can answer,” said Noack. “A six-year-old could press buttons on that poll. Other than that, many people don’t answer them.”

In 2013, 26 per cent of the Canadian population refused to fill out the census. The census is hand delivered to people’s doors, and surveyors will call each resident to help them complete the forms.

Even less people will answer an automated call from a polling agency, which forces the pollster to make more calls or decrease the sample size.

What do polls really tell us?

Polling provides insight into what people are thinking.

“Without the polls, we have no idea who is winning or losing,” said Bozinoff. “You can’t see an opinion, but it is out there, and it can be measured to a very exact degree or margin of error or percentage. There is no other way to get that information.”

Grenier said that people like the clarity of data. The problem is that people only remember the mistakes—the times when the polling agencies get it wrong—and that can make them mistrustful of future results.

“You focus on the bad news versus the good news all the time.”

This article originally appeared in print on Wednesday, October 22, 2014.

Comments are closed.

Previous Next
Test Caption
Test Description goes like this
Read previous post:
Game changer: Rye is now the place to study sports media

Ryerson is now the place to study for sport media enthusiasts hoping to score big in the industry. In September,...