The DOBlog

Tag: The-Business-of-IQ

  • Conferences and me for the end of 2007…

    Conference season is upon us in the Information Quality Community…

    At the end of September I’m off to Las Vegas to deliver a presentation at the IAIDQ’s North American conference the IDQ 2007 Conference.

    At the end of October I’m off to sunny London for the IRMUK Data Management and Information Quality Conferences. This will be my sixth year at this conference and my fourth as a presenter. This year I hit the ‘big leagues’ with a 3 hour tutorial on some of the legal aspects of Information Quality, going head to head with Larry English (amongst others)on the time table.

    Then in November the Irish CoP of the IAIDQ, the IQ Network will be hosting our IQ Forum… we’re planning it to co-incide with World Quality Day on the 8th of November to tie in with some IAIDQ events that will be taking place world wide.

    Who knows, maybe I’ll meet somebody from Dell at one of those conferences who might be able to fix my laptop problem before Christmas. 😉
    That would be nice.

  • Mobile phone registration

    The Irish Government have again trundled out a proposal to force mandatory registration of pre-paid mobile phones. It is stated that this will be a wonderful weapon in the war on drugs, organised crime and pixies.

    There are two small problems with the proposal as it currently stands.

    1. It is unlikely to work as the politicians claim it will
    2. It is unlikely to work as the politicians hope it will

    Now, technically, this is just one problem but it is such a doozy I thought it would be worth mentioning at least twice.

    The reason it is unlikely to work as the politicians claim it will is that in order to ensure that the Register of Mobiles does not become filled with Michael M. Ouses or I.P Freelys the process will require some form of validation of name and address. In order to mitigate the risk of forged or fraudlent documentation being used (which would result in Mr Freely freely getting his fone phraudulently phone fraudulently) this documentation will need to be of some ‘official’ form.

    For Bill Pay phones the usual documentation required is a passport or drivers licence (a work ID on its own is not usually sufficient in my experience) and a utility bill – these prove you are who you say you are and you live where you say you live. In order for the Register of Mobiles to meet the stringent evidentiary requirements that the stated purpose require (to deter criminals using mobiles and to assist in tracking and apprehending them via their mobile records) then a very high level of validity and verifiability will be required of the information used for identification purposes when the phone is being purchased.

    The majority of Ready-to-go customers would seem to be children and teenagers (I’ve lost track of the number of phones my teenage sister-in-law has had over the past few years). They may not have a passport, are unlikely to have a driver’s licence (until they are in their late teens) and are extremely unlikely to be in possession of a valid utility bill in their name.

    So how will they register their phones? Are the Government proposing that the phones would be registered to the parents of these children and teenagers? What then if the child is involved in some criminal activity? Does the parent become a suspect because of the mobile phone records?

    “I didn’t beat up little Johnnie… I’d left my phone at home. My ma hates Johnnie – she must’ve done it!”.

    One solution proposed to this (which according to the Irish Daily Mail came from Civil Servants in a review of this idea update– thanks to Antoin at eire.com who has a blog post which quotes the Dept of Communications on this topic) would be to implement the Register of Mobiles only after a National Identity Card was introduced in Ireland. In theory, this would give a standardised, State-backed identity (and possibly a unique identifier for the person). However there are no current proposals to implement such a card and previous proposals have met with opposition from various quarters.

    A further issue is that name and address data ages over time. People move house, get married, get divorced or die. What mechanism will the Government require to ensure that the information registered on the Register of Mobiles is maintained accurately and in a form that meets or exceeds the evidentiary requirements of the legal system? This is not an issue for bill Pay phones as if the bill ceases to be paid the phone is cut off… and if the bill is still being paid but the address is no longer valid the Authorities have other investigative avenues open to them (such the payment records). For ‘ready-to-throw’ mobiles this is and will be a critical problem. Not only can the person move address, but the phone may ‘move person’ by being swapped, loaned or shared between family members (this may happen with bill pay phones but it is reasonable to assume that in the case of a bill pay phone the ‘sharing’ or ‘lending’ of the phone would be temporary).

    Currently we have a very important national register of people which is collated and maintained for a very serious and important function in the operation of the State. It’s called the Electoral Register and, not to put too fine a point on it, it has some ‘issues’ with the quality of the information there-in and the levels of duplication and inaccuracy in the data. What confidence should we have that the Government will have learned the lessons of the Electoral Register in the design and implementation of any new Register of Mobiles?

    I am not saying we should not require registration of pre-pay mobile phones (all operators currently encourage registration through ‘free credit’ bribes). However if we are to require citizens to give up elements of personal privacy and provide information about their mobile phone usage to State Agencies then it is essential that the system work as it is intended it will and that the information captured meets or exceeds the expectations of the politicians, the police and, most importantly, the citizen. Crucially this must happen with out the information captured being excessive or irrelevant to the stated purposes for registration.

    If we require people to provide information into a system and set of processes that will eventually degrade into an unmanaged cacophony of inaccurate, incomplete, inconsistent and otherwise just plain awful data rather than a symphony of polished, reliable and policed information then we will have achieved nothing other than a layer of paperwork and a burden on mobile phone operators and their customers. Those with criminal intent will pervert the system – foreign SIMS, imported phones, stolen phones etc.

    By definition they don’t play by the rules.

  • IQ Trainwrecks.com

    The IAIDQ, an organisation I’m involved in for Information & Data Quality, has launched IQTrainwrecks.com as a resource to collate and comment on media stories highlighting the cost and impacts of non-quality information.

    Here’s an example of one of their posts. Our Electoral Register features there as do a few other stories. The IAIDQ team hope that this site will evolve into a focal point for informed comment on the real costs and impacts of poor information quality.

  • Electoral Register issues

    Astute followers of the recent General Election in which there were a number of reports of problems with our national electoral register will doubtless be wondering where my comments on that issue might be.

    Rest assured that I haven’t forgotten about it and am working on collating the media reports of issues and tracking down other substantiated cases of problems with the electoral register. I will be producing an updated analysis of the likely root causes which I will publish here and over at the IQ Network website (www.iqnetwork.org). I may even get around to doing a presentation on it to the IQ Network in the coming months.

    Suffice it to say the issues are both simple and complex and the likely scope of root causes ranges from a failure of governance from Government, the lack of a clear strategy for improving the quality of the register, a reliance on scrap and rework to ensure accuracy (doomed from the beginning) and also the actions or inactions of key people in the voter registration and verification processes (including the public).

    Updates to come soon.

  • First Post Anniversary… Electoral Register Processes Still Broken

    The DobBlog is 1 year old today.

    What have we achieved? Well, about 50,000 words on the state of the Irish Electoral Register, syndicated publication of an article based on those words in two International newsletters for IT/Business Intelligence professionals and a wordpress template I’m finally not unhappy with.

    The electoral register is still buggered (that’s a technical term). The scrap and rework (as predicted) was inconsistent and hasn’t fixed the underlying problems. As soon as the clean up stopped, the register has begun to drift to inaccuracy again.

    The Government continues to be cavalier about the issues involved in our electoral system… a number of constituencies will be under represented in the next Dail because of the failure of the Government to react to the population changes in the Census. That’s assuming the election can go ahead given the Constitutional challenge that has commenced.

    The understanding of the importance of good quality information and well designed processes to gather and use that information has grown however amongst a small (and growing) group of occasional visitors to the blog.

    Hopefully the next 12 months will bring enlightenment to Government on some of the issues I’ve blogged about this year and perhaps they will seek out good practices. Hopefully as well we will see some critical commentary in the media on these types of issue. I lost count of the number of pieces I submitted to Irish media during the year. My hit rate outside of Ireland this year is 2 for 2… domestically it is 0 for lots more.

  • Scrap & Rework Article

    Many moons ago I posted a piece on this blog about Information Quality Scrap and Rework in the Irish Electoral Register. This article was submitted to a number of Irish newspapers at the time (when it was very topical) and was referenced at length by tuppenceworth and others.

    Earlier this year I was invited to write for Larry English’s column in DM Review magazine, an international trade magazine for Information Management and Business Intelligence. It appeared in the on-line ‘extended edition’ of the magazine. Here is the link to this month’s DM Review… I’m on page 5 (I’m credited as a contributor but the content appeared here first, and was picked up by B-Eye-Network last year also. For real afficianados of Irish Electoral Register issues, here’s a link to the paper I wrote on the issue back in the dim and distant past.

    Thanks to Larry and his staff for helping with a minor re-write to make the article more ‘American-friendly’. (Larry’s profile that I’ve linked to above is from the IAIDQ website – www.iaidq.org)

  • Count down to an Information Quality clash?

    Daylight Savings time starts in the US on the 11th of March – that’s next week. DST doesn’t start in Ireland or the UK until the 25th of March. The US change comes about under an Energy Protection Act passed last year.

    Microsoft are warning people in the US that their PCs won’t automatically update (not that mine ever did) and are assuring people that VISTA already handles it.

    So what will happen if your PC has the incorrect locale settings (data)? Will that have triggered it to download the various patches for Windows and Outlook? Have European firms checked that they have no dependencies on US daylight savings time in other software or calculations?

    I’m probably fretting over nothing but seemingly innocuous base data can, if not managed correctly, have a big impact on business processes and on people’s lives.

    My advice – check your locale settings even if you’re using a Mac.

  • Electoral Reg (A slight return)

    OK. In an attempt to make this interesting to the kids, I’m ripping off Jimi Hendrix lyrics.

    The Sunday Business Post reported over the weekend that up to 170,000 people may have been taken of the Electoral Register in error. Apparently politicians of all hues are trading war stories of bungled clean ups on the electoral register. Apparently, amongst other things, entire housing estates have been taken off the register and dead people have resurrected and re-registered to vote. The Minister in question, Dick Roche, has even had to acknowledge that he knows of an incident of a disappearing housing estate in Wicklow (wouldn’t it be ironic if it was up on Turlough Hill muses the author, mixing his Irish geography).

    The Fianna Faíl TD for Meath, Johnny Brady, has commented that:

    • huge numbers of elderly voters have been removed from the Register in his area
    • no letters were left to inform people they were being taken off the Register (or at least people don’t recall getting such a letter).

    According to Mr Brady “Some of the field officers who called to houses decided that if they were not at home, they were taken off”. This suggests a degree of inconsitency in the approaches between local authorities… in my earlier post on when the people came knocking I pointed out that they hadn’t spoken to me, but as of today I’m still on the Register. Therefore it would seem that different rules are being applied in Meath and Wexford.

    Divergences in work practices in maintaining the Register is one of the contributing root causes to the whole original mess. Anecdotes of Local Authorities using the Obituraries in the local and national papers to identify dead people were mentioned in dispatches not so long ago.

    And the treatment of the dead is clearly one of the key root causes for the original shambles… with 30% of Waterford’s voters being members of the daisy pushing brigade. Of course, this discrepancy is matched by the inconsistency between the numbers on the register now and the population as measured by the Census.

    Good grief. What a mess.

    Way back in the summer I wrote that the proposal to rebuild the register by going door to door would not address the actual deficiencies in the register. The key approach should have been to tackle the root causes – such as wildly varying work practices in different local authority areas and then to push out cleansing of the register. This should have been done in a clear and transparent manner.

    However, at this point it is important to bear in mind that often the answer you get to a problem isn’t necessarily the answer you want. The Opposition parties seem to have had an expectation that there would be no collateral damage in the clean up of the register. A cliche involving eggs, omlettes and breakage springs to mind. Rather than engage in debate based on anecdote the Opposition parties should try to ‘speak with data’ and to identify clear examples of where people have been taken off the register in error and get evidence of what process or inaction on the part of the Minister or Local Authorities lead to the error.

    For example, Gay (Gabriel) Mitchell (Fine Gael TD) reported his personal experience where he wrote to the Local Authority officers responsible for the Register to tell them that there were two people resident at his address with the same name (his son is also called Gabriel). However only one Gabriel Mitchell was left on the Register. Why? Did Deputy Mitchell forget to include the respective dates of birth? Did his letter fall through the cracks?

    Dick Roche attempted to clean the register by running en masse a broken process. Throwing people at it to perform door to door checks did not address key root causes (like the fact that you can’t change your name on the Electoral Registration form – it only allows for changes of address). When you throw into the mix that the door to door checkers:

    1. Don’t hang around long enough to talk to people (in my personal experience)
    2. Call during the day when people are at work (might that explain why entire estates in the commuter belt of Dublin have disappeared off the Register?)
    3. Seem to have an inconsistent practice as to how to deal with people who don’t answer the door

    then this whole process is a phenomonal white elephant that may have served to make a bad situation slightly worse.

    However – with regard to people who have been taken off the Register in error… there is a question of personal responsibilty here. If they wish to be registered to vote then they should check the register at their local libraries or Garda stations or online (if they are in the 50.7% of people who have internet access) and get themselves registered.

    If you are not in you can’t win. If you’re not on the register you can’t complain about the government you get. And by my reckoning that’s what we have at the moment.

  • Please buy Expedia an Atlas…

    Following on from Michel Neylon’s on-going battle with Amazon, it looks like the illness has begun to affect Expedia (who may need to buy an atlas from Amazon).

    A colleague of mine just tried to book a hotel room in London for a weekend away. She got her itinerary number and had confirmed availability and price and was trying to give her credit card details to pay for the booking.

    On Expedia, you have to tell them if you are a UK address or a non-UK address (I suspect that this is to present different address format templates). My colleague selected “Non-UK” and proceeded to fill in her address details.

    Until she got to the part where they wanted to capture Country. Ireland wasn’t listed. Neither was Éire, Republic of Ireland, Irish Republic or Southern Ireland (all common alternatives that are sometimes used).

    Nepal and the South Mariana Islands were available options though. Lucky for them.

    Let me put it another way… the drop down list of countries was significantly incomplete for a company that is operating within the European Union (25 states and counting). Ireland hasn’t been part of the UK since 1922.

    My colleague rang Expedia to find out what was going on and to see if the order could be completed over the phone. To her surprise she was told that “expedia can’t take orders from Ireland”. Which is the equivalent of “the computer says no” from Little Britain.

    I wonder if the legal eagles who hang out over at tuppenceworth would have an opinion on the legality of Expedia’s business model, which to my mind smacks of an unjustified (and unjustifiable) restriction on free movement of services within the European Union and the European Free Trade Area.

    In the mean time, my colleague will be using a different site to book her accomodation in London. Until, of course, “the computer says no”.

    (editor’s note: I’ll stick the links ‘n’ stuff into this later).

  • Propogation of information errors and the risks of using surrogate sources

    ….ye wha’?

    There has been a lot written in relation to the electoral register and other matters about using information from other sources to improve the quality of information that you have or to create a new set of information.

    This makes sense, other people may already have done much of the work for you and, effectively, all you need to do is to copy their work and edit it to meet your needs. In most cases it may be faster and cheaper to use such ‘surrogates’ for reality to meet your information needs than to go to the effort of going to the real-world things (people, stock-rooms where ever) and actually starting from scratch to build exactly the information you need in the format you require to exactly your standards and formats.

    There is, however, a price to pay for having such surrogate sources available to you. You need to accept that

    1. The format and structure of the information may need to be changed to fit your systems or processes
    2. The information you are using may itself be innaccurate, incomplete or inconsistent.
    3. If you are combining it with other information, it will require investment in tools and skills to properly match and consolidate your information into a valid version of the truth.

    These risks apply to organisations buying marketing lists to integrate with their CRM systems but also could be applied to students relying on the Internet to present them with the content for their academic projects or journalists trawling for content for newspaper articles or reviews.

    Recurrence of common errors, phrases or inaccuracies in term papers is one way that academia has of identifying academic fraud. Similar techniques might be applied in other arenas to identify and track instances of copyright infringement.

    In businesses dealing with thousands of records, the cost/risk analysis is relatively straightforward. The recommendation I would make is that clear processes to manage suppliers and to measure the quality of the information they provide you based on a defined standard for completeness, consistency, duplication, conformity etc. is essential. Random sampling of surrogate data sources for accuracy (not every 100th record but a truly random sample) is also strongly recommended.

    These are EXACTLY the same techniques that manufacturing industries use to ensure the quality of the raw material inputs to their processes. If it works for industries where low quality can kill (such as pharmaceuticals), why shouldn’t work for you?

    For students, journalists and those of us hacking away in the blogosphere the recommendation is simple. Only rely on surrogate sources if you absolutely have to. If you use someone elses work as your source, credit them. If you don’t want to credit them then make sure you verify the accuracy of their work either by actually verifying against reality or by checking with at least one other source.

    That way you avoid having the errors of your source become your errors also and you don’t run the risk of someone crying foul and either suing you for stealing their copyright (and copyright does apply to content posted on the internet and in blogs) or taking whatever other sanctions might apply (such as kicking you off your college course).

    In many cases the costs and effort involved in double checking (particularly for a once of piece of writing) are neglibily different to the costs of actually starting from scratch and building your information up yourself. And, depending on the context, it may even be more enjoyable.

    The New York Times not so long ago had to relearn the lessons of checking stories with at least one other source for accuracy.

    Horatio Caine in CSI:Miami always tells his team to “trust, but verify”.

    When using surrogate sources for real-world information in any arena you must assess the risk of doing so and put in place the necessary controls so that you can trust that you have verified.

    (c) Daragh O Brien 2006 (just in case)