However it is often difficult to get an organisation to rectify all the Data Quality issues, before they embark on delivering reports and information to the business users who need it.

One of the interesting sessions at next years SAS Users Conference in New Zealand is by Zeeman van der Merwe who is talking about the work he is doing at ACC.  I had the pleasure of meeting Zeeman a while ago and to talk to him about his project and he is definitely taking the recommended approach of sponsorship from the top and covering areas such as Data Governance, Data Stewardship and Data Quality reporting.

One of the Data Warehouse projects we are working on has a sister project dealing with Data Quality.  It is fair to say we that we have yet to get the organisation to fully understand the impact data quality has on the business and the necessity to rectify the issues.  Everybody does of course agree there are a lot of issues with the quality of the data which is a good start.

I always remember in my presales days at SAS the words customers always uttered “yes we have major data quality issues” shortly followed by “but we don’t have any money to pay to fix them”.

Anyway on this project we are lucky enough to have SAS Enterprise Data Integration Server at our beck and call and so have the ability to use Dataflux on the Warehouse data.  So we have done a number of tactical Data Discovery and Data Validation pieces of works.

So far we have completed:

• Validation of Phone Numbers
• Validation of Addresses
• Customer/Person matching

The Phone Number validation was the first one we did and we picked it as it was a discrete piece of work we could time bound, while we worked through the process to use Dataflux.  We are now looking to close the loop by updating the augmented phone number data Dataflux produced inot the source system, and changing the business rules in the source system to rectify some data entry issues we identified.

I really recommend the idea of picking something small to start out with.

We are now looking at how we productionise the Data Quality routines into out standard Data Warehouse load and reporting processes.  So far the options (in 9.1.3) look like:

• Purchase the full use Dataflux Integration Server
• Schedule Dataflux routines to run on a PC
• Manually run the jobs
• Rewrite the Dataflux jobs in SAS DI Studio

Interesting thing to note is that in SAS 9.2 the Dataflux Integration Server component is bundled in eDI so you can just deploy the Dataflux Architect jobs and run them in your Warehouses standard process flows.

We still havent decided whihc option will work the best, but are thinking it is going to be the DI Studio option in the interim as consistency and stability of loads is one of our major focuses.

I have to say I love Dataflux and all that it does (I even believe the Dataflux team now have a stringer presence in the development of SAS Data Integration Server under the “Project Unity” banner).

I note that Dataflux jumped to the top of the Gartner Magic Quadrant in 2009.  I always struggle to find this when I need them, so here are the Data Quality ones for 2008 and 2009.

Gartner 2008 Data Quality Magic Quadrant

Gartner 2009 Data Quality Magic Quadrant

I agree with him when he says that he struggled with the definition of Metadata as “data about data”, so I really like his version of:

“When it comes to data, does the user know what it means?”

Victor takes us through a walk through of cleaning up address data using Dataflux, with screenshots and all!

I am always amazed at how many companies recognise they have a data quality issue, and then relegate it to a ‘spare time’ project, rather than a priority.

As they say “you can’t manage what you don’t measure” just as important is my own “any decision based on crap data is likely to be a crap decision”