On Monday the 16th January 2012 the Irish Examiner ran a story that purported to have found that 93% of the Irish public “decried” the decision of the Minister for Foreign Affairs to close Ireland’s embassy in the Vatican City State. The article detailed how they had undertaken a review of correspondence released under the Freedom Of Information Act which showed that 93% of people in Ireland were against the closure. To cap it off, the article was picked up in the Editorial as well.
Except that that isn’t what they had uncovered. The setting out of the statistics they had found in the sensationalised way they presented them was a gross distortion of the facts. A distortion that would, to paraphrase Winston Churchill, “be half way around the world before the truth had its boots on”).
What they had uncovered is that of the 102 people who wrote in to the Minister for Foreign Affairs about the issue, 93% of them expressed a negative opinion about the closure. The population of Ireland is approximately 4.5 million people. 95 people is closer to 0.000021%. While I may not have the academic qualifications in Mathematical physics that my famous comedian namesake has but I know that 95 people (that’s 93% of 102) is slightly less than 93% of the Irish public
Or, to put it another way, significantly and substantially below the statistical margin for error usually applied in political opinion research by professional research companies.
Or to put it another way, over 99% of the population cared so little about the closure of the Vatican Embassy that they couldn’t be bothered expressing an opinion to the Minister.
Of course, the fact is that there were letters written about this issue. And the people who wrote them were expressing their opinion. And 93% of them were against the closure. Â In fact, in defending themselves on Twitter against an onslaught of people who spotted the primary school maths level of error in the misuse of statistics in the article, the Irish Examiner twitter account repeatedly states that (and I’m paraphrasing the actual tweets here slightly) “for clarification we did point out that the analysis was based on the letters and emails”. But it is inaccurate and incorrect to conflate the 93% of negative comment in those letters to the entire population as the sample size is not statistically valid or representative being
- Too small (for a statistically valid sample of the Irish public you would need between 384 and 666 people selected RANDOMLY, not from a biased population. That’s why RED C and others use sample sizes of around 1000 people at least for phone surveys etc
- Inherently biased. 93% of cranky people were very cranky is not a headline. The population set is skewed towards one end of the distribution curve of opinion you would likely find in the wider population.
Then today we see a story in the Examiner about how Lucinda Creighton, a Junior Minister in the Dept of Foreign Affairs is backing a campaign to reopen the embassy because
there’s a very strong, and important and sizeable amount of people who are disappointed with the decision and want to see it overturned and who clearly aren’t happy
What? Like 93% of the Public Lucinda? Where is your data to show the size, strength, and importance of this group? Have you done a study? What was the sample size?
As a benchmark reference for what is needed for an Opinion Poll to validly represent the opinions of the Irish Public, here’s what a reputable polling company says on their website:
For all national population opinion polls RED C interview a random sample of 1,000+ adults aged 18+ by telephone. This sample size is the recognised sample required by polling organisations for ensuring accuracy on political voting intention surveys. The accuracy level is estimated to be approximately plus or minus 3 per cent on any given result at 95% confidence levels.
Anything less than that is not statistically valid data and can’t be held out as representing the opinion of the entire public.
As an Information Quality Certified Professional and an active member of the Information Quality Profession on an International level for nearly a decade I am ethically bound to cry “BULLSHIT!!” on inaccuracies and errors in  information and in how it is presented. The comments from Ms Creighton are a good example of what that is important in the Information Quality and wider Information Management profession. If bullshit analysis or analysis based on flawed or inherently poor quality data is relied upon to make strategic decisions then we invariably wind up with bullshit decisions and flawed actions.
And that effects everything from conversation with family, chats in the pub, business investment decisions, political decision making, through to social policy. Data, Information, and Statistics are COOL and are powerful. They should be treated with respect. People publishing them should take time to understand them so that their readers won’t be mislead. And care should be taken in compiling them so that bias does not skew the results.
So, having had no joy or actual engagement from the Irish Examiner on the issue I forwarded my complaint to the Press Ombudsman yesterday pointing out that the article would seem, based on the disconnect between the headline, the leading paragraph, and the general thrust of it, to be in breach of the Code of Practice of Press Council of Ireland.
I just hope they can tell the difference between lies, damned lies, and fudged statistics. (This Yes Minister clip about Opinion Polls shows how even validly sampled ones can be biased by question format and structure in the survey design).
Pingback: Correction from Irish Examiner re: Vatican Closure | The DOBlog