It has been fascinating to read the recent critique by the Picker Institute of NHS England’s “friends and family test” (FFT).
The FFT, as you will recall, was announced by the Prime Minister in January 2012, and is claimed to be “a single measure that looks at the quality of care across the country”.
But the Picker Institute, an independent charity, claims that new research shows that “the ‘simple, headline metric’ used for the test does not provide a reliable basis for comparing services or identifying those performing best.”.
Oddly enough, this finding was presaged by Tim Kelsey, the NHS England director in charge of the test, who noted in November 2013 that the FFT “should not be viewed as a statistic but as a ‘line of sight measure’”.
But while noting the FFT’s failure as a measure of quality or performance, Chris Graham, director of research and policy at Picker, argues that it may still have merit as an improvement tool – not because of the scores, but because of the patient comments which are collected at the same time.
The power of stories
Of course, this is where our ears prick up. We have argued – and indeed demonstrated – for many years now that stories told by patient and carers can be an important stimulus to learning and improvement. The stories on Patient Opinion not only identify issues requiring action, but often identify solutions too. And, in connecting with the heart as well as the head, many stories are powerful motivators of change.
There seems to be implicit acceptance by NHS England of this, since each recent announcement of new FFT results has been supported, not by any demonstration of the power of scores, but by numerous examples of the power of stories.
If it is true that the purpose of the FFT policy is no longer about measuring, but about improving, and the means of such improvement is not scores, but the stories patients tell, then some immediate questions come to mind:
- Is the score needed at all? Or would the stories prompt improvements just as well without any scores?
- If what prompts improvement are actionable stories, then is it important (was it ever?) to insist that 100% of patients are asked to respond? After all, many people will have little they want to say.
- Is it OK to collect and act on stories from patients, but not let the contributors know the difference they made? Especially in a context where many feel that giving feedback is pointless, as “nothing will be done”.
Could FFT link to Patient Opinion?
This last question leads to another: if FFT turns out to be a system for collecting stories and delivering them to staff, how does it differ from Patient Opinion?
Part of the answer to this is that Patient Opinion is:
- public - everyone can read everything
- open - anyone can post, which means that we can hear from those who lack access to care, as well as those who gain access
- personal - you get something back from your story: a response, maybe a resolution, even a service change
- independent - we’re outside the system
- inclusive - providers, commissioners, healthwatch, patient groups, CQC and others are all in the loop
Of course, Patient Opinion isn’t as big or well-resourced as FFT. But perhaps we could have the best of both worlds? What if all patients were asked the “FFT question”, and those who had more to say were able to just “click through” to Patient Opinion, where they could raise concerns, offer praise, and resolve issues in real time?
We think this approach could add real value for patients and staff alike, at very little cost. In fact, we’re already piloting exactly this model with Healthcare Communications, a major provider of FFT surveys and analysis.
If you’re interested in learning more, or have a view about the value or purpose of FFT, do comment below, or get in touch.
Where next for the Friends and Family Test?
Where next for the Friends and Family Test? https://patientopinion.blob.core.windows.net/profile-pictures/75f71c7a-7f9c-4b02-9116-1ed4ff616c90.jpg Care Opinion 0114 281 6256 https://www.careopinion.org.uk /content/uk/logos/co-header-logo-2020-default.pngQuestion from Care Opinion
Posted by James Munro, Chief executive, Care Opinion, on
Response from Peter Denton, Healthwatch Manager, Healthwatch Tameside on 10 Jun 2014 at 15:24
Fascinating blog James. Lots of things in there I have been saying for a while. The Net Promoter Score methodology used by FFT just seems plain bonkers. There is not enough choice about where people can go for healthcare (especially in an emergency) to make it meaningful - for example, how many banks serve the area covered by one A&E department?
The question “How likely are you to recommend our [ward/A&E department/maternity service] to friends and family if they needed similar care or treatment?” simply has too many variables in it that are nothing to do with the quality of care (e.g. location of service relative to where your friends and family live). There are too many assumptions that people need to make in order to answer it - therefore any purely statistical analysis is likely to be flawed.
But yes - we have always said locally that it provides an opportunity to gather rich narrative from patients. If FFT provides an opportunity to open a conversation with someone you previously wouldn't have talked to, it's got to be a good thing. We'd love it to plug into Patient Opinion because it means the public and all partners have access to the raw data - and for feedback to be given clearly in the public domain.
That said, the important thing for us is that the narrative is read, analysed, acted on and feedback given. The specific mechanism for that feels less important than the facility and commitment to do it.
Response from Shaun Maher, Improvement Advisor, Person-Centred Collaborative, Healthcare Improvement Scotland on 10 Jun 2014 at 21:06
Hi James
Thanks for raising these important issues.
I've never been a fan of FFT and there was also an earlier piece of work published by Picker in 2012 (commissioned by the CQC) to identify the best way to ask an overarching experience question. Their recommendation was categorically NOT to use the FFT and they offered a more context appropriate suggestion that had evaluated well in their research. For some reason this advice was ignored.
I'm a strong believer in open and transparent forms of feedback focused on improvement and I think PO is an invaluable ally in working towards this goal. I also think that we have (and continue to) undervalued feedback as a driver of improvement and an indicator of quality.
But, I think we need some kind of measurement over time that helps us to know whether the changes we make actually lead to improvement - something that can be a bit of a challenge (although not impossible) with narrative type feedback.
I do think there is opportunity to augment / complement the work of PO and other similar resources further with an overarching question that could be widely asked or sampled across the health and care system. I think an open qualitative question that simply asks: "What was good? What do you think could be improved?" (asked either as a compound or two separate questions) could generate some very useful improvement ideas alongside the more detailed stories such as those on PO.
It would also be possible to generate a bit of quantitative data by using sentiment analysis to quantify positive and negative comments. You could then look at this data over time as a means to identify whether your changes were leading to improvement. If you were getting things right positive comments should increase (or at least stay the same) and negative comments should decrease. You would also have the opportunity to look at the emergent themes, see some read across to PO themes, and hopefully see the problem themes diminish as you set about your improvement work.
Enough rambling for now and thanks again for your blog.
All the best
Shaun
Response from James Munro, Chief executive, Care Opinion on 10 Jun 2014 at 22:13
@Peter, thanks for thoughtful response. I agree, the question is odd and - as Shaun points out below - was also the subject of a critical Picker study.
I agree with the emphasis on actually reading and acting on feedback - and this is why some kind of transparent loop is so important. Why should people keep giving their scores and comments, if they see no result?
@Shaun, thanks for reminding me of the Picker work on the FFT (and other) questions. I don't know why this question was chosen either.
I have no issue with measurement (I was once a quantitative researcher), which is clearly important when we wish to make comparisons by time or place. As you say, we need to measure when we have questions like "well, did we change this?"
But so often in the NHS I hear people say "we need to listen", and then reach for a measuring tool. I think the urge to measure often inhibits the ability to listen - and certainly, does little for the experience of "being listened to", which is what patients and carers value so highly.
In short, we need to be clear on our purposes.
Thanks again for such helpful comments :)
Response from Ian Reeves on 13 Jun 2014 at 21:13
I've used PO a few times as a test earlier this year. I had some reponses that made me aware of where the system wasn't performing as well as it should, and some stories where patients reminded us where we were doing ok.
Our ward has been using FFT for a month. We have a 'score' as do all the other wards, but there's little understanding of this score by the staff, the variation between the wards or - in any sense - the score's meaning.
I asked staff if they handed the cards to the families of patients who had died, but they didn't. We're missing out on a crucial group who's experience we should want to know, but who probably can't fill out a FFT card at the time they leave the ward.
We have comments from FFT, which have been scanned from the cards and emailed back to the ward (ours are positive this time) but there's no chance for us to respond directly, and no easy way to make this information publicly available.I've used PO to post positive comments about an outpatient service, our FFT is limited to wards only. How can FFT be as flexible as PO, when our care is delivered across so many different wards and specialties?
Overall, I'd much prefer PO to FFT. I'm not sure what the relative cost differences are, does anyone have any data on that?
Response from James Munro, Chief executive, Care Opinion on 15 Jun 2014 at 09:25
@Ian
Many thanks for sharing these experiences from the front line. If FFT is intended as a tool for improvement, as well (or instead of) measurement, then we need to hear more experiences like this.
There are clearly some limitations to FFT, from your point of view, such as the lack of any ability to communicate or share actions. But then, there are limitations to Patient Opinion too.
The question of costs is also important and I’m not aware of any published estimates of the costs of implementing FFT across a healthcare provider, though there must be data available. Some (rather sketchy) cost assumptions were made by government before the policy was rolled out, and are available here:
I’m not sure it would be right to make direct cost comparisons between FFT and PO since we do very different things in different ways. Having said that, the price of PO to a healthcare provider is typically £11k or less per annum at the moment.
Of course, you get less “data”. An interesting question is whether you get less improvement.
Response from Ian Reeves on 15 Jun 2014 at 14:25
I'm not sure that PO does gives less data. It's less 'detailed' but narrative trumps numbers nearly every time.For example, assume a ward (1) has a FFT 'score' of 56.
Ward 1 56Ward 2 83Ward 3 47Ward 4 62Ward 5 33
Ward 1 also has 4 comments from FFT , but these are all 'one liners' such as 'the staff were lovely and wonderful care, thank you'
With this 'data' ward 1 appears to be in the middle of the pack relative to peers, and has some positive comments.
For the wards what's missing and crucial, is any information that would help ward 1 improve it's care.
For managers what's missing is there's no indication of if ward 1 is really 83-56= 27 FFT 'points' 'worse than ward 2, or if it's just because ward 2 is mostly day case surgery, while ward 1 has more patients who require complex care. Clearly, over time this might work better, but the sample size and frequency of sampling need careful thought if the system is to achieve a balance between working well enough to detect variation, but not overwhelm.
For the 'friends and family' what is missing is any acknowledgement that their comment has been heard. This is particularly relevant if the comment highlights where care could be better, they've gone to the effort (and perceived risk of retribution) in order to make their comment known. The service should acknowledge this, and reassure them that their comments are welcome, and change will occur as a result.
In contrast, PO allows a comment on any aspect of the service, not just ward based, so car parking and signs are recognised as important too, not just clinical care. It allows the ward staff and managers to respond, and change, to the issues that really matter enough for someone to provide unsolicited comment, and for patients, (or their friends and family) to know their comments are heard.
Personally, I'd rather listen the voices of my patients than be emailed our average FFT 'score'. I think it's true -as you said in your blog response, that the very act of measuring 'inhibits our ability to listen'.
Perhaps this is more like quantum physics after all?