Why evaluations should start earlier and finish later

In order to answer these questions well, a good evaluation needs to start early enough and finish late enough. “Early” means, ideally, at least 2-3 months before the service change of interest has actually begun. “Late” means, again ideally, after a long enough period to permit a statistically and clinically significant change to be identified – the fewer the numbers using the service, the longer this is likelier to be.
The benefits of an early start are:

1. This allows for time for the service and the evaluators to fully and clearly understand exactly what is intended to change, and what its impact is intended to be – before the change begins to happen. Our experience is that both may require clarification. For example, what is meant by an “integrated” service – what exactly is intended to be “integrated” and between which teams or agencies? And whilst impacts may be described in general terms, the particular metrics for patient experience, or health and social outcomes may be less clear – and thought may not have been given to any potential or expected differences in impacts between different groups of patients.

2. Once the service changes and their hoped-for impact are clear, we can then agree in detail the evaluation questions – what it is that our evaluation is hoped to show – again, before the change begins to happen.

3. If (1) and (2) are done in time, it is then possible to conduct a cycle of baseline interviews with stakeholders. Whilst it may (usually) be possible to retrieve historic activity or financial data, interview perspectives, if not gathered before a service change begins, are lost forever. Knowing clearly how services worked before the change, and what are seen as its strengths and weaknesses, provides very useful context against which to assess subsequent experiences.

We are sometimes asked to report on evaluations within a few weeks or months of a service change being implemented. Longer will usually produce a more reliable result. The benefits of thinking a little longer are:

1. Implementation is very often delayed. The processes of recruitment, financial agreements, multiagency agreements for complex changes, and then take-on of patients very often take longer than was originally hoped. There is a risk of trying to draw conclusions before anything much has happened.

2. Some effects take a long time to be detected. Many current initiatives are based on complex service models, with many elements having to be established and then learn to work and to gel together. There is a risk that something can be considered not to have worked before it has had time to prove itself – or to have worked before that can be confidently concluded.

3. Drawing conclusions about causality is always difficult in evaluations. An impact may have been seen, but to what extent can we be confident that it can be attributed to the change we have made? The more data, the more we can draw reliable conclusions.

If decisions about long-term finance or service organisation are to be based on evaluation findings, we are all responsible for ensuring those decisions are as robustly-based as possible. So, if you’re contemplating an evaluation or a service you provide or commission, we’d be delighted to hear from you; we have both experiences and expertise to share in a wide variety of service evaluations. And when you’re thinking about the timeframe for your evaluation, wherever possible, it may be improved by starting earlier and finishing later.