Why evaluate?

Why evaluate? – James Fitton, Partner

Organisations routinely introduce new things. These things can be a service model, a project, a programme, or an organisational structure. Why they introduce those new things can vary enormously. Sometimes it’s simply because they’ve been told to by someone else; sometimes local champions have successfully argued for a change; sometimes there’s very clear evidence that the new approach will be an improvement – but often the evidence is much murkier.

This is the core argument for evaluation.

Evaluation is a systematic process intended to enable judgement of something’s merit, value or importance. It should stand firmly on the two feet of both evidence and values – quantitative and qualitative data to support that judgement, and a values-driven understanding as to why particular metrics are chosen.

A successful evaluation should therefore enable us to understand:

• Does it work?
• Why does it work?
• How well does it work?
• What difference does it make?

It may also answer questions such as:

• Does it work better for some people than for others? Who and why?
• What should be done to improve it?
• What should be done to secure any benefits in the longer term?

The arguments for evaluation may therefore seem fairly straightforward. These questions are all ones we’d often like to see answered. We don’t let new drugs or therapies be introduced without proper research demonstrating their effectiveness and value – why would we implement a new service model without any checks that it works as intended? After all, “innovative” usually also means “untested” – and shouldn’t we test if something does what its designers claim?

But sometimes things aren’t that straightforward. Here are the five challenges we most commonly hear as to why something can’t or shouldn’t be evaluated:

1. It’s not really possible – you can’t do a randomised controlled trial for everything

We agree! It’s important not to confuse evaluation and formal clinical research, such as RCTs. Some real-world changes have no obvious “control” or “counterfactual” – and the logistics of changing part of a team or part of an organisation may be unethical or impossible. But you can still evaluate – decide what success will look like, and gather statistics and experiences to test if it’s being achieved.

2. It takes too long – we just want to get on with things

Evaluation needn’t delay implementation of changes at all. Many evaluations are designed to be “formative” as well as “summative” i.e. providing information and findings during the course of implementation of change, with the explicit intention of improving the change. The summative element, at the end of the evaluation, will then shape the service or project’s long-term future.

3. It’s too expensive

Yes, independent evaluation isn’t free. Depending on a project’s scale and duration, costs can vary from “not too bad” to “quite a lot.” But this cost needs to be set against the investment in the project or service being evaluated.  Even a relatively modest service can easily cost £1M a year for 10 years. Only 1% of that budget could fund a substantial evaluation –  how much is it worth to make sure it’s implemented optimally? And that it’s achieving what was hoped for at the outset?

4. Our situation is very different – there’s no point trying to draw generalised understandings from what we’re doing

Maybe. But evaluation isn’t primarily about creating new generalised knowledge. It’s about improving what you’re doing, here and now. Some evaluation reports can be genuinely helpful in other places: “Here’s what we did, here’s what we learned, here’s how you can learn from our successes and avoid our mistakes.” Some are very specific to a set of local circumstances.

5. We can do it ourselves – we have metrics, dashboards, we know how we’re doing anyway

Great! This can make an independent evaluation simpler (and cheaper). But it doesn’t remove the value of a fully independent perspective:

• Many staff and other stakeholders feel more comfortable giving fully open and honest feedback to a third party – who can then anonymise their contributions in a report
• There’s always a risk of optimism bias in purely internal evaluations. People who have invested time, thought, and energy into an idea really want it to succeed. Sometimes this blinds people to ways in which what they are doing could be improved
• Independent analysis gives confidence. Evaluation reports are often used in making decisions about long-term support, or about funding of services. Purely internal reports typically have less impact than ones offering independent assurance.

Evaluation is therefore a key tool in the process of service improvement. A well-designed evaluation will support you in thinking through what you’re wanting to achieve by a change; it will provide feedback to improve the process of change; and it will provide assurance to others that you’ve taken the right decisions in the right way.

If you would like to know more about Niche and the evaluation services we provide – contact us via the ‘Get in touch’ option on our site.