The Leaving Cert exam fiasco

So. The Irish Government (in the form of the Dept. of Education and the State Exams Commission [SEC]) are faced with a €1million bill because an exam Superintendent inadvertently distributed the wrong exam paper earlier this week.

An avoidable root cause for this now unavoidable expenditure seems to be that the packaging that exam papers comes in is too similar. The SEC issued a reminder to Superintendents about this very issue. 

Reminders and warnings are ultimately reactive in nature. They scream “we know there is a risk of a screw up here, so be careful now”. They do not, unfortunately, in themselves reduce the risk of the screw up happening – that requires the person receiving the warning to remember in all cases to act on it.

Warnings just give the people who issue the warnings the scope to say “we told you to be careful” as they fire the person who made the error. They are, in effect, a verbal (or written) form of inspecting a defect out of a process before it reaches the customer. They do not improve the process.

So, what might process improvement here be that actually contributes to a reduction in the risk of significant financial loss to the State because one person in one exam centre makes one mistake?

When assessing whether it is worth changing a process, we need to assess the cost, impacts and risks involved. The risk of the wrong exam papers being given out is not that high. However, the cost and impact when it does happen is proving to be significant.

If we assume that the risk of it happening is no more than five times in 100 years then that is a 5% risk each year that something will go wrong (remember – we are dealing with probablity, not a schedule).  We can assume that in any year it happens, as soon as it does everyone involved will be acting on every warning given to make sure it only happens once – the survial instinct kicks in.

 If we assume that the basic financial cost each time will be in the region of €1 million, that means that, prudently, we should see what sort of change can be implemented for an ‘insurance premium’ of €50,ooo  per year. This does not, of course, factor in the reputational damage to government agencies, the PR damage for the elected Minister, the stress impacts on students and their families as exams are rescheduled etc and any potential legal liabilities that might arise. For the sake of argument, we will assume that the monetary equivalent of those risks is  €20,000.

So. What change can we implement for up to  €70,000 per year that would prevent unintentional and indavertent confusion of exam papers because of similarities in their packaging?

One option would be to colour code the packaging with distinct colours (i.e. avoiding orange and brown and sticking with strong bold colours that definitely look different). Use different coloured packaging for each subject for example, or put a coloured line or cross on the packaging. Print a logo on the front of the packaging that illustrates the subject (a book for English, a globe for Geography, Einstein’s head for Physics, a picture of Peig for Irish). Anything to provide a standardised visual clue as to what the subject is.

My preference is for totally colour coded envelopes… If it is Red it is English, Green Irish, Blue French etc. 

Of course, to do it for ALL the subjects offered in the Leaving Cert in ALL centres might prove more costly than the notional €70,000 we’ve set aside as our insurance premium.

This is where we would need to further refine our view of the impact of the risk per subject. For example, investing in coloured wrapping for English is a no-brainer. It is a core subject that everyone does.  Accidentally leaking that paper affects ALL students in EVERY exam centre. That’s what costs the €1million we are trying to avoid paying out 5 out of every 100 years.

Colour coding Classical Studies however might be harder to cost justify. It’s not taken by that many students, it’s not examined in that many exam centres. The cost of colour coding the exam script envelopes for subjects like this could possibly be more than the cost of rescheduling the exam. Also, many of these less taken subjects are examined towards the end of the exams window… further reducing the risk of confusion as the box of exam scripts will be emptying fast.

So. How much would it cost the State Examinations Commission to colour code the top 10 subjects by number of students and number of exam centres? Would we even need 10 subjects coded in this way?

While there is little that can be done to ‘risk proof’ against an intentional leaking of an exam paper other than to have a second (or third) version of the exam on stand-by and having criminal sanctions for people caught doing so, there are simple changes that could be made to risk-proof against accidental leaking.

The only question is does the cost of introducing a preventative control that improves the quality of information presentation (by adding an additional cue – in this case colour) out weigh the risk and impact of having packages that are so similar that they can be accidentally confused. 

What sort of insurance premium against that risk is the SEC willing to pay?