The DOBlog

Tag: data quality

  • Information Quality – Do we have an app for that?

    A few weeks back I got a new iphone. I’d resisted for years, enjoying the pleasures of Nokia and Symbian and the challenges of Palm and Windows Mobile 6.1.

    The fun part for me of any new mobile phone purchase is playing with the new toy  tool and seeing what it can do that my old one couldn’t. For example, back in the 1990s when I did my first upgrade from my first mobile phone (an ericsson model so old that I actually can’t find it referenced on the internet), I found that the new phone was so much smaller and lighter I was actually able to carry it around.

    The irritation I have is when it comes to moving my contacts and synchronising with my various other technologies that hold contact details (laptop, gmail, company address book). Inevitably I wind up with duplication and triplication of contacts. I thought I had the problem licked on the iphone though as there are a number of apps available for managing contact details and reducing duplicates.

    However, having spent a few days using them I am unimpressed as they seem to be making a the traditional rookie mistake in de-duping records – assuming that name matching is enough.

    My brother and father share a given name and a family name. They have different middle initials, different addresses, different phone numbers, different email addresses (all the stuff that you would have in a contact record on your phone). Each application I tried decided that they were a duplicate entry and merged the records. This was annoying.

    In other cases, I have duplicate entries with varying degrees of record completeness. For example, my friend Cathal exists at least 4 times, with one entry having most of his contact details,  with spurious email addresses or social networking nicknames in the others.  The “data quality tool” very kindly merged all the records into the entry that had the least amount of data, and deleting the other records.

    Right now I’m considering firing up talend, datanomic, or informatica tools to dedupe a dump from my iphone and reload it to the phone, and then hopefully that will cascade through the rest of my data stores when I synchronise.

    But I’ll need to draw a data flow map of all of that to make sure.

    Grrrrhhh.

    So. If the existing tools for data quality on the iphone are not up to the jobs, what is missing? The good news is that the data sets are fairly clearly structured (once they get into the iphone), so that is less of a concern than the actual processing of matching and consolidation of records.

    1. Probability scoring across multiple fields would be nice. If two people have the same name but significantly different contact details then it is very probable they are not the same person. A corollary – if there are two records with the same name and one has contact information and the other record has only a name, chances are they are duplicates.
    2. Presentation of matches for review. While the machine can make good guesses where the name and contact details are the same, where there is confusion, the matches should be flagged for a review by the phone user (the “Data Controller”). This way we can avoid having to unpick erroneous matches.
    3. Merging of records should be done on a more structured basis, with mapping of fields being user-customisable based on a standard template. I despair of important contact information being dumped into a notes field (it reminds me too much of when I had to try and migrate data out of a Siebel call centre system a few years ago).
    4. The matching should be able to cater for multi-lingual input (as phones don’t all live and work in english speaking lands).

    There may be other requirements that I am not thinking of here at the moment, but those 4 are a starting point. Perhaps an obliging Data Quality tool vendor will develop an iphone app to a web service for matching contact records.

    Personally, I think that having such a service available would help raise awareness of the value of quality non-duplicated contact information to individuals and to organisations.  However, the app on its own isn’t enough as the average smart-phone user may have personal information held in a variety of places and, just like in a large enterprise with lots of data stores, creating a “Single View of Contact” will require you to understand the flow of your contact information around your tools (i.e. does the phone update the laptop and does the laptop synch to google apps and does google apps synch to the phone?) to avoid the cleanup work being undone the next time you plug your phone into your PC.

    Information Quality Management poses challenges for the enterprise, but can also create friction for the individual trying to manage something as simple as a list of contacts across multiple information stores.

    Do we have an app for that?

  • Who then is my customer?

    Two weeks ago I had the privilege of taking part in the IAIDQ’s Ask the Expert Webinar for World Quality Day (or as it will now be know, World Information Quality Day).

    The general format of the event was that a few of the IAIDQ Directors shared stories from their personal experiences or professional insights and extrapolated out what the landscape might be like in 2014 (the 10th anniversary of the IAIDQ).

    A key factor in all of the stories that were shared was the need to focus on the needs of your information customer, and the fact that the information customer may not be the person who you think they are. More often than not, failing to consider the needs of your information customers can result in outcomes that are significantly below expectations.

    One of my favourite legal maxims is Lord Atkin’s definition of who your ‘neighbour’ is who you owe legal duties of care to. He describes your ‘neighbour’ as being anyone who you should reasonably have in your mind when undertaking any action, or deciding not to take any action. While this defines a ‘neighbour’ from the point of view of litigation, I think it is also a very good definition of your “customer” in any process.

    Recently I had the misfortune to witness first hand what happens when one part of an organisation institutes a change in a process without ensuring that the people who they should have reasonably had in their mind when instituting the change were aware that the change was coming.

    My wife had a surgical procedure and a drain was inserted for a few days. After about 2 days, the drain was full and needed to be changed. The nurses on the ward couldn’t figure out how to change my wife’s drain because the drain that had been inserted was a new type which the surgical teams had elected to go with but which the ward nurses had never seen before.

    For a further full day my wife suffered the indignity of various medical staff attempting to figure out how to change the drain.

    1. There was no replacement drain of that type available on the ward. The connections were incompatible with the standard drain that was readily available to staff on the ward and which they were familiar with.
    2. When a replacement drain was sourced and fitted, no-one could figure out how to actually activate the magic vacuum function of it that made it work. The instructions on the device itself were incomplete.

    When the mystery of the drain fitting was eventually solved, the puzzle of how to actually read the amount of fluid being drained presented itself, which was only of importance as the surgeon had left instructions that the drain was to be removed once the output had dropped below a certain amount. The device itself presented misleading information, appearing to be filled to one level but when emptied out in fact containing a lesser amount (an information presentation quality problem one might say).

    The impacts of all this were:

    • A distressed and disturbed patient increasingly worried about the quality of care she was receiving.
    • Wasted time and resources pulling medical staff from other duties to try and solve the mystery of the drain
    • A very peeved and increasingly irate quality management blogger growing more annoyed at the whole situation.
    • Medical staff feeling and looking incompetent in front of a patient (and the patient’s family)

    Eventually the issues were sorted out and the drain was removed, but the outcome was a decidedly sub-optimal one for all involved. And it could have been easily avoided had there been proper communication about the change to the ward nurses and the doctors in the department from the surgical teams when they changed their standard. Had the surgical teams asked the question of who should they have in their minds to communicate with when taking an action, surely the post-op nurses should have featured in there somewhere?

    I would be tempted to say “silly Health Service” if I hadn’t seen exactly this type of scenario play out in day to day operations and flagship IT projects during the course of my career. Whether it is changing the format of a spreadsheet report so it can’t be loaded into a database or filtered, changing a reporting standard, changing meta-data or reference data, or changing process steps, each of these can result in poor quality information outcomes and irate information customers.

    So, while information quality is defined from the perspective of your information customers, you should take the time to step back and ask yourself who those information customers actually are before making changes that impact on the downstream ability of those customers to meet the needs of their customers.

  • Software Quality, Information Quality, and Customer Service

    Cripes. It’s been a month since I last posted here. Time flies when you are helping your boss figure out how to divide your work up before you leave the company in 3 weeks. I’ve also been very busy with my work in the International Association for Information and Data Quality – lots of interesting things happening there, including the Blog Carnival for Data Quality which I’ll be hosting come Monday!

    One of the things I do in the IAIDQ is moderate and manage the IQTrainwrecks.com website. It is a resource site for people which captures real world stories of how poor quality information impacts on people, companies, and even economies.

    Earlier this week I posted a case that was flagged to me by the nice people over at Tuppenceworth.ie concerning double-charging on customer accounts arising from a software bug. Details of that story can be found on IQTrainwrecks and on Tuppenceworth. I’d advise you to read either of those posts as they provide the context necessary for what follows here. (more…)

  • Cripes, the blog has been name-checked by my publisher…

    TwentyMajor isn’t the only blogger in the pay of a publisher (I’m conveniently ignoring Grandad and the others as Irish bloggers are too darned fond of publishing these days. If you want to know who all the Irish bloggers with publishers are then Damien Mulley probably has a list)!

    I recently wrote an industry report for a UK publisher on Information Quality strategy. The publisher then swapped all my references to Information Quality to references to Data Quality as that was their ‘brand’ on the publication. I prefer the term Information Quality for a variety of reasons.

    As this runs to over 100 pages of A4 it has a lot of words in it. My fingers were tired after typing it. Unlike Twenty’s book, I’ve got pictures in mine (not those kind of pictures, unfortunately, but nice diagrams of concepts related to strategy and Information Quality. If you want the other kind of pictures, you’ll need to go here.)

    In the marketing blurb and bumph that I put together for the publisher I mentioned this blog and the IQTrainwrecks.com blog. Imagine my surprise when I opened a sales email from the publisher today (yes, they included me on the sales mailing list… the irony is not lost on me… information quality, author, not likely to buy my own report when I’ve got the four drafts of it on the lappytop here).

    So, for the next few weeks I’ll have to look all serious and proper in a ‘knowing what I’m talking about’ kind of way to encourage people to by my report. (I had toyed with some variation on booky-wook but it just doesn’t work – reporty-wort… no thanks, I don’t want warts).

    So things I’ll have to refrain from doing include:

    1. Engaging in pointless satirical attacks on the government or businesses just for a laugh, unless I can find an Information Quality angle
    2. Talking too loudly about politics
    3. Giving out about rural/urban digital divides in Ireland
    4. Parsing and reformatting the arguments of leading Irish opinion writers to expose the absence of logic or argument therein.
    5. Engaging in socio-economic analysis of the fate of highstreet purveyors of dirty water parading as coffee.
    6. Swearing

    That last one is a f***ing pain in the a**.

    If any of you are interested in buying my ‘umble little report, it is available for sale from Ark Group via this link.. . This link will make them think you got the email they sent to me, and you can get a discount, getting the yoke for £202.50 including postage and packing (normally £345+£7.50p&p. (Or click here to avoid the email campaign software…)

    And if any of you would like to see the content that I’d have preferred the link in the sales person’s to send you to (coz it highlights the need for good quality management of your information quality) then just click away here to go to IQTrainwrecks.com

    Thanks to Larry, Tom, Danette, the wifey for their support while I was writing the report and Stephanie and Vanessa at Ark Group for their encouragement to get it finished by the deadline.