Time saved is not the same as productivity gained

New technologies – in particular, AI scribes based on ambient voice technology (AVT) – promise to free up clinical time and increase NHS productivity. AI scribes automatically transcribe and summarise clinical consultations, with the intention to reduce the documentation burden for clinicians.

Grand claims are made for the ability of AVT to improve NHS productivity, for example within the 10 Year Health Plan for England. However, the methodology to evaluate the productivity impacts of new interventions is poorly developed, causing fundamental uncertainties for researchers.

For instance, how should we define and measure clinical productivity? What study designs can robustly capture changes in how staff allocate their time? How do we distinguish genuine efficiency gains from task displacement or quality trade-offs?

To start addressing this gap, THIS Institute and the Health Foundation convened a workshop with around 40 delegates (including health services researchers, health economists, clinicians, programme managers, and improvement leads). This blog post summarises what we heard.

Delegates split across five breakout groups, each facilitated by a member of the organising team and tackling a different methodological question:

• conceptualising and measuring productivity gains
• study design and time-use measurement
• distinguishing genuine efficiency gains from task displacement
• assessing long-term and unintended consequences
• understanding patient and staff experience

What we heard

Time released is not the same as a productivity gained. This was the most consistent message of the day, raised independently by several groups. If a technology frees up 20 minutes, that time might be used in different ways, for example toward more patients seen, longer consultations, training, service improvement, or simply a lighter workload. These are different outcomes with different values that a metric counting only appointments won’t distinguish.

Ask who benefits before asking what was saved. Evaluations should establish whether patients, staff, or the wider system benefit, and follow effects across the whole pathway rather than stopping at the individual clinician. Benefits may show up as avoided repeat appointments; costs may show up as work quietly transferred elsewhere. A bottleneck further down the pathway can absorb an efficiency gain entirely.

The comparator deserves more attention. AVT will inevitably change the allocation of work between existing human-technology systems. It will impact the role of Dictaphones, medical secretaries, existing transcription software, and typed notes. Measured against a well-run dictation service, the marginal gain may be modest, and comparators need to be defined more consistently.

New work is created as well as removed. Studies of AI scribes typically count the time clinicians spend checking and editing notes within documentation time. But there may be work that moves to other parts of the care process. AI-generated notes that are longer than conventional notes, or that bury the key points, can be harder for others to interpret. A colleague reading the record later may have to work harder to establish the clinician’s assessment of risk or what they intended to happen next.

Implementation costs (e.g., infrastructure, IT support, governance) are often left out of the sums, and under-resourcing here limits what the technology can deliver. Work can also shift onto patients and families, for instance time spent reviewing AI-generated summaries in the NHS App.

When used in isolation, every method for measuring time use has limitations. The study-design group weighed electronic health record (EHR) audit logs, direct observation, self-report, time-recording apps, video capture, and patient-recorded data. EHR logs are appealingly automatic, but most systems weren’t built for research. Records are left open for hours and timestamps are unreliable. Direct observation captures interactions and nuances of technology use rather than just timestamps.

Although behaviour might change under observation, over time, participants tend to revert to usual behaviour, which can be tracked in the data. Self-report needs motivated participants, and if study framing is interpreted as surveillance, the consequence may be refusals or distorted data. Video analysis could become efficient once automated but raises ethical and consent questions. Across every method, effort is not the same as elapsed time, and mental load is poorly captured by all of them.

The long-term risks are not primarily about time. The unintended-consequences group raised the erosion of critical thinking where AI drafts the record, distinguishing de-skilling in experienced clinicians from ‘never-skilling’ in trainees who may never acquire the skill at all.

Others worried that AI will privilege measurable biomedical outcomes over psychosocial care, and that AI-mediated triage will favour patients who already know how to work the system, widening inequalities. There was also scepticism about ‘AI’ as a catch-all term: different AI technologies carry different risks (e.g., machine learning versus LLMs), and evaluations should say precisely what is being assessed.

Qualitative work is not optional. The group made the case for focus groups across career stages, ethnography that surfaces the hidden work of making a technology function, and deliberative methods. Involvement has to be designed in from the start of a study, not added later, and reaching marginalised groups matters most with technologies that risk leaving people behind. Existing patient feedback channels could be adapted for this, and developers could be asked to build feedback routes into the tools themselves.

Emerging recommendations

No group settled on one preferred method. Instead, a shared framework emerged: combine quantitative measures of resources, activity, and outcomes, with evidence on workflow change, experience, and unintended consequences, then triangulate across them.

Candidate measures included staff time and workload, access and waiting times, clinical outcomes and consultation quality, patients seen and needs resolved, and duplicated tests avoided. These should be reported alongside guardrails covering clinical safety, patient experience, equity, staff wellbeing, and professional accountability.

Suggestions for further methodological work included tracking licensing-exam performance to detect training effects, using routine primary care records to spot shifts in prescribing and referrals, peer review of note quality, regular horizon scanning for unintended consequences, testing existing implementation frameworks against AI products, and automating video analysis.

Where next?

Since the workshop, NHS England has announced an accelerated AI rollout (£10bn over three years), expected to deliver around half the commitments in the 10 Year Health Plan, much of it resting on AVT and a new AI triage tool for the NHS App. The evidence base underpinning this ambition is therefore still underdeveloped, and it will be critical to build this further.

That makes the absence of agreed evaluation methods a practical problem, not just an academic one. If we cannot distinguish a genuine gain from a displaced task, we will not know whether the investment has paid off — and may not find out until an unintended consequence is entrenched and hard to reverse.

We are grateful to everyone who joined the session. THIS Institute and the Health Foundation are continuing to work in this area and would welcome contact from researchers interested in taking any of these threads further.

Authors

• Niels Peek – THIS Institute
• Matthew Woodward – THIS Institute
• Ross Goldstone – The Health Foundation
• Alison Moulds – The Health Foundation
• Stephen Rocks – The Health Foundation
• Emily Froggatt – The Health Foundation
• Lukasz Zielinski – Barts Health NHS Trust