Let me tell you a story on robust supplier appraisal

On Tuesday we learnt that some of our education policy thinking appears to be shaped by what looks like unreliable evidence - the sort of approach which looks selective as opposed to robust. It is ironic that we want more useful research as an outcome of universities yet in trying to achieve that outcome we are using, what some might refer to as 'Mr Men evidence'. The sad thing is that whether or not you agree with the Education Secretary's views, his credibility has now been weakened, whereas a more robust approach would have strengthened his case.

Gove has also weakened his argument for education reform and could be perceived as hypocritical when he discusses critical analysis. We need to have an education system which encourages more critical analysis, particularly for the next generation of procurement practitioners. We need practitioners who can look at comparative research and evidence, identify weaknesses and then take an informed view - we need procurement practitioners skilled in critical analysis, particularly in the area of supplier appraisal.

Let me give you an example. I recently had sight of a supplier appraisal which considered suppliers past performance. References were requested. Assuming the supplier was able to provide three reference sites of delivering similar work, they passed!  No effort seemed to be expended in validating the references. Reliable and robust evidence?

While I used to think asking a supplier for three references sites was a nonsense, based on the assumption that no one would be daft enough to provide a bad reference site. That was until I came across a supplier who did just that and was surprised when in the debriefing I fed back the bad reference. But how can we be sure a past customer doesn't bare some sort of grudge and disproportionately reflects the poor elements of what the supplier thought was an excellent service. Reliable and robust?

My view is that we need to ask for a comprehensive list of all the similar contracts provided, over say  three years, including their values.  Then, subject to the risk associated with the procurement, select a  random sample and interview those who can speak with authority on the past performance. It may not be completely robust but it is significantly better than the 'Mr Men' approach some practice.

Of course one of the easiest ways of improving supplier appraisal is having good historical records of the quality of service your own organisation has received in the past. Time and time again i have seen poor past performance being overlooked and only 'remembered' when Tender responses are received. Organisational memory has a role here as has easily accessible and reliable records. Here too we find a bizarre example from the Education Secretary, who, in his evidence to Wednesday's Common's Education Committee, advocated a radical new approach to ''O' levels' which replaces the existing A, B, C, etc. grades with a numerical system. An 'A* grade' would therefore be spilt to become 'Grades 1 and 2, 'A grade' would be spilt to become 3 and 4, etc.  We would also see greater focus on end of course exams and the demise of coursework. By gove, isn't that the system under which my own 'O levels' operated in 1973? If only we had really good history lessons things may be improved - no, I don't mean in the education system, I mean in supplier appraisal.