A mixed-methods study of challenges experienced by clinical teams in measuring improvement

Published in


Woodcock T, Liberati EG, Dixon-Woods M. A mixed-methods study of challenges experienced by clinical teams in measuring improvement. BMJ Quality & Safety. 2021;30:106-115 http://dx.doi.org/10.1136/bmjqs-2018-009048

  • 13 March 2023


Why it matters

Although it is widely accepted that measurement is a vital part of improving quality in healthcare, how well it’s done varies considerably.

Difficulties in measurement can mean that improvement projects don’t have reliable or valid data to assess progress, report accurately, or produce useful learning. Problems with data collection and interpretation include missing data or lack of clarity in sampling strategies.

One issue is that when measures are set externally, for example by regulators or funders, clinicians may feel the measurements are not relevant to things that concern them. But not much is known about what happens when clinical teams choose their own measures and design their own data collection systems.

Our study set out to understand the challenges clinical teams face when undertaking measurement based on measures they have chosen themselves.

Our approach

We drew on the independent evaluation of a patient safety improvement programme using an approach called Safer Clinical Systems, which seeks to support organisations to develop capacity to detect and address weaknesses in their systems and to measure and report their improvement.

Teams taking part in the programme across nine UK hospitals were expected to develop a detailed plan setting out measures for collecting useful data; establish data collection systems; and analyse and interpret their data.

Our study explored the experiences of the teams who took part in the programme by interviewing team members and observing their programme-related activities. We combined this with an expert review of the measurement plans and an analysis of the data collected.

What we found

Teams found setting up systems for collecting data was time and resource-intensive and was something they struggled with – sometimes having to compensate by doing extra, unpaid work.

Some teams identified too many measures. Not all plans demonstrated understanding of the underlying principles needed to gather good quality data. Many operational definitions of improvement measures needed to be more specific to capture improvements, while some were not logically linked to improvement actions. Teams were often confronted with missing data or measurement systems not being used in the way they were intended.

Data analysis also varied in quality, linked to skills and capacity in the teams. Poor quality data hindered meaningful analysis.

Our study suggests that measurement is a highly technical task, requiring specific expertise and skills. Teams without such expertise faced many difficulties and tended to underestimate challenges involved in measurement and the amount of time needed.

Brief training is unlikely to be enough to upskill teams adequately. Improved access to validated measures would be helpful, but more structural initiatives and programmes that build capability for measurement are also needed.

Read the full article

Licensed under Creative Commons

These symbols show that the contents of this page are published under a Creative Commons licence called CC-BY-NC-ND 4.0.

It means that you’re free to reuse this work. In fact, we encourage it. We want our research to reach people who can help improve quality and safety in healthcare. But we do have a few rules:

  • Make sure you acknowledge The Healthcare Improvement Studies Institute (THIS Institute) as the creator and link back to this webpage.
  • You can’t sell this work for a fee, or use it for any activity that generates revenue without our permission.
  • Please don’t distribute a modified version to others without our permission.

You can read the fine print about the licence on the Creative Commons website. It’s meant to help us keep the integrity of our work and stay true to our values.

But ultimately we want our work to have impact. So if you’ve got a use in mind but you’re not sure it’s allowed, just ask us at enquiries@thisinstitute.cam.ac.uk