Lies, damned lies, and statistics

Of course, the fact is that there were letters written about this issue. And the people who wrote them were expressing their opinion. And 93% of them were against the closure.  In fact, in defending themselves on Twitter against an onslaught of people who spotted the primary school maths level of error in the misuse of statistics in the article, the Irish Examiner twitter account repeatedly states that (and I’m paraphrasing the actual tweets here slightly) “for clarification we did point out that the analysis was based on the letters and emails”. But it is inaccurate and incorrect to conflate the 93% of negative comment in those letters to the entire population as the sample size is not statistically valid or representative being

1. Too small (for a statistically valid sample of the Irish public you would need between 384 and 666 people selected RANDOMLY, not from a biased population. That’s why RED C and others use sample sizes of around 1000 people at least for phone surveys etc
2. Inherently biased. 93% of cranky people were very cranky is not a headline. The population set is skewed towards one end of the distribution curve of opinion you would likely find in the wider population.

Then today we see a story in the Examiner about how Lucinda Creighton, a Junior Minister in the Dept of Foreign Affairs is backing a campaign to reopen the embassy because

there’s a very strong, and important and sizeable amount of people who are disappointed with the decision and want to see it overturned and who clearly aren’t happy

What? Like 93% of the Public Lucinda? Where is your data to show the size, strength, and importance of this group? Have you done a study? What was the sample size?

As a benchmark reference for what is needed for an Opinion Poll to validly represent the opinions of the Irish Public, here’s what a reputable polling company says on their website:

For all national population opinion polls RED C interview a random sample of 1,000+ adults aged 18+ by telephone. This sample size is the recognised sample required by polling organisations for ensuring accuracy on political voting intention surveys. The accuracy level is estimated to be approximately plus or minus 3 per cent on any given result at 95% confidence levels.

Anything less than that is not statistically valid data and can’t be held out as representing the opinion of the entire public.

As an Information Quality Certified Professional and an active member of the Information Quality Profession on an International level for nearly a decade I am ethically bound to cry “BULLSHIT!!” on inaccuracies and errors in  information and in how it is presented. The comments from Ms Creighton are a good example of what that is important in the Information Quality and wider Information Management profession. If bullshit analysis or analysis based on flawed or inherently poor quality data is relied upon to make strategic decisions then we invariably wind up with bullshit decisions and flawed actions.

And that effects everything from conversation with family, chats in the pub, business investment decisions, political decision making, through to social policy. Data, Information, and Statistics are COOL and are powerful. They should be treated with respect. People publishing them should take time to understand them so that their readers won’t be mislead. And care should be taken in compiling them so that bias does not skew the results.

So, having had no joy or actual engagement from the Irish Examiner on the issue I forwarded my complaint to the Press Ombudsman yesterday pointing out that the article would seem, based on the disconnect between the headline, the leading paragraph, and the general thrust of it, to be in breach of the Code of Practice of Press Council of Ireland.

I just hope they can tell the difference between lies, damned lies, and fudged statistics. (This Yes Minister clip about Opinion Polls shows how even validly sampled ones can be biased by question format and structure in the survey design).