Blog: Evidence Based Decision Making – where’s the evidence?

What’s the old hierarchy? Data can contain information, which can represent knowledge, which can be applied as wisdom? (Bearing in mind that for each ‘can’ there is an implicit ‘may not’).

So where does evidence fit into this? I would argue that it belongs more at the wisdom end of the hierarchy rather than at the data end. Please don’t interpret this as saying I don’t like data – I am an engineer after all, I love the stuff! But data needs to be interpreted and applied for it to have any value, and that involves judgement.

In fact data by itself can be a dangerous thing. For several years now I have been facilitating decision processes using multi-criteria models with stakeholder groups. Early on, observing and learning, I saw an example where experts had brought along a lot of data and analysis; the other stakeholders (including end-users, funders and engineers) were very pleased to see this data and implicitly assumed we would then use that data to make the decision. They were keen to move on and get it done. My mentor on this occasion who was leading the facilitation pushed back: we had a simple decision model at that stage, so we examined that model and what it told us about a) the customer requirements, and b) the available decision outcomes. It quickly became apparent that the data, although no doubt valuable for other purposes, was not relevant to the decisions we had to make. So what did we do? We agreed that we were not going to get new data that would help our decision in the time-frame we had available to us. However, we did have the full range of experience and knowledge (wisdom) in the room to be able to make a set of judgements, that everyone felt comfortable led to an appropriate decision. The logic and assumptions were recorded to explain and defend any future challenge.

This wasn’t a one-off. This particular project involved a whole set of workshops covering different but related decisions, and different individual stakeholders. Often there was a mismatch between the data available and the decision that needed to be made. If data was available, then there was inevitably a groundswell of opinion in the room that this data should be used to make the decision – the implication being that it was better to use the wrong data than to resort to judgement. Judgement was perceived as ‘subjective’ and in some way a weak or a bad thing.

I have been looking out to see if some ‘cultures’ are different in this regard than others. It seems to me that scientists and engineers should understand the relationship between data and relevant information, but my experience suggests they are if anything worse, even more liable to be seduced by data. I suspect a lot of current management training and culture doesn’t help; I remember the dictum ‘what you can’t measure, you can’t manage’. Well the problem with that is that if you only manage the things you can measure (ie with objective, fact-based data), you are probably only managing a small subset of the problem space.

From experience what is needed is process, data and judgement – all 3. The process tells you what judgement and data is needed; the data can be used to inform judgement; and the process then needs to enable collaboration and make value judgements explicit. A software model can help to structure the decision space, then capture the relevant data and judgements. Subjectivity becomes a positive thing, as diverse perspectives are explored and converged where appropriate, or the differences clarified and understood.