Thursday, 7 February 2013

It's time for procurement policy outcome evaluations

I'm sure many of you followed the discussion on the evidence given by Colin Cram and Jon Hughes to the Public Administration Select Committee. We had a good critique, over a few Spendmatters' Blogs and the readers associated comments, then we had Colin's pieces in Public Leaders and Supply Management once again with associated comments. We need more discussion on how procurement is approached and more debate. But yet I feel there was a fundamental weakness in the whole discussion - it lacked any rigour and solid evidence.

There is an additional weakness though in relying on the wisdom of procurement's eminence grise to provide the right answers - while those in the procurement world can debate good practice, their wisdom is cloaked, in spite of the obvious disagreements, in 'the narrowness of the same school'. By way of contrast, we know that the Prime Minister takes advice from the philosopher Nassim Taleb. It is interesting when reading Chapter 12 of Taleb's 'Antifragile' that, had the philosopher been asked to comment on the best structural model for procurement delivery he could have been expected to have given a different view. Just to be clear, what I am saying is that, even though there is disagreement amongst some of the more influential procurement commentators, if we want to identify what is best, sometimes it is worth considering the views of those outside our world.      

One way of opening up to others views is through the use of social media. But as we make greater use of social media, and blogs in particular, for sharing thoughts on how to improve procurement we need to be cautious of taking opinions too seriously, regardless of the deference due, in the absence of evidence. I was surprised to discover that The Times differentiates between 'opinion' and 'reporting' through the use of ragged edges - we don't see that differentiation in most blogs. Opinions have a tendency to swing like pendulums. One day a particular view is fashionable and the next it's not.

A useful illustration of the need to be wary of opinions was provided in the second session of oral evidence provided to the same PASC. There we had the unexpected luxury of one of the witnesses, Kevin Craven (Chair of the CBI procurement sub-committee of the CBI PSSB, in answer to Question 55) stating "Generally the MoJ is seen as doing good work in the procurement area", yet on that very same day, MoJ procurement was being described as 'shambolic' by MPs in the same building - need I say more!

The situation isn't helped by
practitioners woeful participation in research - try to complete a robust survey and you find yourself begging for responses. Improving response rates would be helpful all round. Practitioners would learn what is really happening and we could then have useful critical discussion and challenge of whether there may be a better way.

But even what we hear from practitioners may fall short - what we really need is robust impact evaluations. That would help us answer the 'So what?' questions.

So I was both saddened and pleased to learn that Thomas Pringle, a Donegal TD (Irish equivalent of an MP) had the common sense and courage to call for an impact assessment of the Irish Procurement Strategy to centralise:
My sadness was based on the fact that I don't know whether those responsible for the Irish Procurement Strategy will have already completed an impact assessment - we would all benefit from sight of it if they have. But let's also remember that an impact assessment is likely to be no more than an unproven hypothesis with a high degree of 'crystal ball gazing' and anticipation of what 'might happen'. That's different from a reflective outcome evaluation which says 'what did happen'. But that brings a different problem, there may not actually be an independent outcome evaluation elsewhere which could have been drawn upon. So our astute TD from Donegal may well have nothing more than opinions as to whether centralised procurement will lead to job loses or not.

Another option could be to take up some of my suggested lessons for procurement from the CERN scientists approach to getting answers - yet somehow I doubt that we really want answers. 

So as a pragmatic way forward, what we really need are outcome evaluations. Outcome evaluations provide the sometimes uncomfortable truth that what it was hoped would happen hasn't actually happened. That in itself can be very useful if we have lessons which can be learnt from and can then look for other solutions. But there also needs to be caution. I have had visibility of outcome evaluations which were no more than cosmetic - they put a great political gloss on the outcomes achieved so that funders could celebrate and more funding would follow. 

No, we need robust, honest, and critical outcome evaluations - warts and all - based on reliable baselines. That way we can rely less on eloquent rhetoric and opinion and really understand what makes a difference. Such evidence wouldn't just help those responding to Thomas Pringle TD, they would help demonstrate procurement's real contribution. 

No comments:

Post a Comment