A mixed-methods study of challenges experienced by clinical teams in measuring improvement

Why it matters

Quality improvement projects don’t always generate reliable or useful data, and problems in measurement, data collection, and interpretation are often to blame.

These problems can arise from disagreements about which types of measures to prioritise, faulty data collection systems, or when externally imposed measures are perceived as irrelevant or inappropriate, for example. Without useful and reliable data, it can be hard to say whether improvement interventions are successful, and this risks eroding confidence in the evidence base for improvement.

One possible solution could be for clinical teams to measure improvement themselves, choosing their own measures and designing and implementing data collection systems fitted to their own local circumstances. Yet little evidence exists on how well this works in practice.

In this article, we report how some clinical teams have fared using locally selected measures, explore the challenges they faced, and suggest approaches to overcome them.

Our approach

We evaluated a major patient safety improvement programme run in the UK, which used an approach known as Safer Clinical Systems. As part of this approach, multidisciplinary teams of frontline clinicians at nine NHS hospitals took part in the programme between 2011 and 2016.

In this mixed methods study, we used a qualitative approach to study the participating teams’ experiences and perception of measurement, while also reviewing the measurement plans and data collected by the teams for the programme.

What we found

• Improvement programmes that emphasise local ownership and local selection of measures, such as Safer Clinical Systems, may face the same challenges to measurement that others do.
• Clinical teams often struggled to produce a high-quality measurement plan, to provide clear definitions of their measures and data, or to complete data collection and analysis reliably.
• Measurement is a highly technical task requiring a degree of expertise and dedicated staff.
• Brief training interventions and manuals may not be enough to bring most clinical teams fully up to speed on measurement. Building dedicated capability and capacity for measurement may be necessary, and it will take time.
• Accessible repositories of validated measures are likely to be increasingly important in improving measurement for improvement.