Surveys – a snapshot of opinions that can go either way

In the 1936 US Presidential election, Republican Alf Landon was expected to soundly beat Democrat Franklin Roosevelt by more than six percentage points, according to weekly magazine Literary Digest.

Literary Digest was a popular weekly (think Time and Newsweek) then, and surely one would think the weight of its reputation and reach supported the accuracy of its poll.

Moreover, the poll sample size was two million, an astonishing figure even by today’s technological and communication standards. But Literary Digest could not have been more wrong.

Roosevelt defeated Landon by obtaining 61 per cent of the vote, the biggest landslide in US presidential election history. The magazine’s reputation was destroyed, and within 18 months, it went out of business.

On the other hand, that election created the reputation of another poll which accurately forecasted the result based on only 50,000 respondents. The company was run by George Gallup, a name now synonymous with opinion polls in the US.

The problem with the Literary Digest survey is known as sampling bias, where the intended population studied is not adequately represented in the representative respondents. The survey respondents were mostly the magazine’s own readers and automobile owners – a group of overwhelmingly white, rich, and Republican-leaning voters.

In short, the weekly did not collect information from a representative batch of US voters.

Doing a survey today is easy with many free websites offering easy-to-use tools. Want to know where employees want to spend the next family day or what your fellow students think of the cafeteria food? Just send out survey questions to your intended audience and you can quickly get a picture of their general opinion.

It seems simple enough, but what if as the survey designer, you really want the cafe operator to be driven out? You’ve complained to your friends about the food, and they seem to agree with you, and the same group of people form the majority of responses to your survey.

Your survey’s sampling bias can be mitigated by using responses from a wider range of people. But there are still things you can do to ruin the cafe operator. The survey can be designed to have leading questions – most responses will inevitably lead to a negative picture of the cafeteria.

Scientific surveys are not designed to ask every single 30 million-plus Malaysians their personal views. There are not many surveys worth doing if it takes too long to complete and process.

By the time a 30 million-respondent survey is completed, the underlying conditions may have changed – the economy has changed, or new facts have changed people’s minds or people have simply moved on to the next shiny thing.

Pollsters all over the world have been running surveys for decades in business and politics with fairly reasonable results that can be acted on. But as data collection has become ubiquitous, we are constantly being fed it to influence our consumption and opinion.

Data and its analysis are being used widely to justify agendas, pedal products and promote policies and politicians.

Personally, I will always respond to a survey if it can help a student, a reputable research institution or just somebody who wants to find out. Association with a certain group or agenda does not necessarily make a survey dubious – it is just a reminder to take things into perspective, especially if a survey claim sounds outrageous.

A proper pollster should also be able to disclose the survey methodology when required.

Twelve years after making its name in the Roosevelt election, the Gallup organisation predicted that Thomas Dewey would defeat Harry Truman in the 1948 election, by between five and 15 per cent.

Truman actually won the election by 4.5 per cent. Gallup contributed the error to ending his polling three weeks before Election Day.

And herein lies another important thing you should remember about surveys – that it is a snapshot of sentiment or opinion at a particular moment. Things can turn the other way very quickly.

This is the personal opinion of the writer and does not necessarily represent the views of Twentytwo13.

Tagged with: