“Three out of four dentists read Lockergnome”. - Now all that is necessary to validate that statement is to find three dentists who click onto these pages. Of course, it will also be necessary to find that one dentist who is not a Lockergnome reader.

Is this deceptive? - Absolutely. However, it parallels what happens when polls are reported. There are a multitude of reasons that polls are misleading. The following are five points to take into consideration:

• Very rarely is it indicated what the questions asked were. How each question is stated can bias the response.
• The sample size is often not indicated. When the sample size it stated, it usually is too small to be representative of the population. The pollsters, especially in the frenzy of an election campaign, must churn out data. A misrepresentative sample size is as good as having no data at all.
• The means by which the data were gathered are not mentioned. For example, if a poll was done in the Boston area as to who should be the Most Valuable Player (MVP) of this NBA season, Kobe Bryant and Chris Paul would not stand a chance. Location matters and that is usually not mentioned when the results are presented.
• What statistics were used to analyze the data is critical. This kind of close scrutiny would interest a statistics person and it simply is not mentioned. What statistical measures used is important. What confidence levels were used is important.
• The results and methodology are not subject to peer review. A study for a scientific journal, for example, is reviewed by people knowledgeable about the topic and knowledgeable in paradigm design. This includes a good working understanding of the statistics and the interpretation of the statistics. This is usually not done by the polls that are presented in the media.

There are more reasons. The point is that, at best, the polls are a quick snapshot of a passing moment. The validity of the polls may be questioned. The reliability of the polls may be questioned. Is this snapshot representative of the “bigger picture”? - It might be and it might not be. It does fill a need in a twenty hour news culture and it does make the pollsters a huge amount of money. Therefore the public will hear about polling news daily - even if this information is essentially meaningless.

Catherine Forsythe
Director of Operations
FlyingHamster:  http://flyinghamster.com/

[tag]polls, sample size, research paradigm, statistics, peer review, validity, reliability, bias, news[/tag]