Numbers that lie

Last week I commented on the problems associated with a typo in a contract of employment.  In that case a decimal place positioned wrongly led to a court case over an offer of either £2m or £200k.

Then Saturday's Times introduced an new twist to the ongoing investigation into the curious death of a GCHQ mathematical whizz kid.  Ironically the contractor analysing the DNA made a mistake in that "two numbers were put in the wrong way round" on a computer.  As a result somewhere in the region of 18 months investigation into his death were misdirected.

If nothing else these two examples highlight the need quality assurance when dealing with numbers - not just the calculations but the transposition. While the numbers may not have lied they were just the wrong numbers!  Yet as I observe procurement practice I rarely come across that checking even though I have seen mistakes in 'inputting the figures' and the calculations prior to inputting.

However, for a long time I have been concerned about the professions blind acceptance of evaluation matrices.  We appear to assume numbers introduce some form of objectivity while forgetting that many of those numbers are a completely subjective allocation.


Then we have price/quality weighting. My concern here is the subjectivity of the selection of the relative weighting.  What is the probability of a collection of procurement professionals, placed in the same room, given the same procurement problem, agreeing on the relative weighting? For example, how many would agree, if given the same scenario and only the choice of either a 65/35 weighting, a 60/40 weighting and a 70/30 weighting?  This is serious stuff as the financial impact on the buying organisation could be enormous, let alone the impact of loss of business to potential bidders.  Effectively the dice are weighted.

We also appear to have a belief in the received wisdom of 'the price score' based on a percentage of the lowest price is correct. I'd love to know whether there is any sort of scientific or mathematical foundation on which those points can be relied upon?

Finally we have the issue of the integrity of the scoring matrices themselves. I have rarely found a procurement professional who has actually developed and checked their own spreadsheet matrix.  Instead a legacy matrix is passed from one organisation to the next and, like a virus, any inbuilt mistake in the formulae infects the procurement decisions in the recipient organisation.

So I have a 'number' of concerns about whether or not numbers can lie:

1. Have the figures been checked/quality assured to ensure there are no typographical errors?
2. Has the inherited scoring matrix been check for internal arithmetical accuracy?
3. Are there procedures in place to check and challenge the rationale behind quality/price weigthings?
4. Is there any scientific or mathematical basis for accepting the current percentage based allocation of price scores?
5. Are procedures in place to test the robustness of the subjective allocation of 'quality' scores?

I think this is an area where CIPS could add value in commissioning research and providing guidance before we end up with one of the core tools of procurement being discredited.

P.S. Just in case anyone didn't realise, and it appears some didn't, yesterday's post was my feeble attempt at an April Fool!  For those taken in I would suggest an urgent need to follow yesterday's final lesson learnt, "take out adequate insurance to protect against any claims of gullibility".

Background reading:
Hamilton, F. (2012) 'Dead spy's family points to 'dark arts' of espionage as false DNA trail is revealed', The Times, 31 March, p.3