Sense about Science ? equipping people to make sense of science and evidence
News and Comment
Parent-led tool shows value of public engagement
21 June 2016Website shares complex children's heart surgery data with families
Sense about Science has been working with researchers and patients’ families to develop an innovative website, Understanding Children’s Heart Surgery Outcomes. The website, which launches today, will help people make sense of published survival data about children’s heart surgery in the UK and Ireland. The researchers who developed the site, are calling for the end to an era of confusion and alarm about children's heart surgery statistics. Transparency without accessibility is not enough: stats must be put in context, they say.
The website, shows decision makers and parents that, because hospitals treat different patients, hospitals should not be ranked by their survival rates — high performing hospitals can have lower survival rates simply because they are taking on the most complex cases. An individual hospital’s actual survival rate should only be compared to its own predicted range, which is determined by the complexity of the procedures it undertakes, among other factors. The website also sets out why if a hospital's survival rate is below its predicted range, it need not indicate alarm, but rather serves as a trigger for further investigation.
The website was developed by Christina Pagel from University College London and Sir David Spiegelhalter from the University of Cambridge, in collaboration with Sense about Science and experimental psychologist Tim Rakow from King’s College London. It explains a risk adjustment method known as PRAiS (Partial Risk Adjustment in Surgery).
“This website draws a line under an era of poor risk communication of hospital surgery statistics,” said Tracey Brown, director of Sense about Science. “In 2013 over-interpretation of faulty data resulted in temporary hospital closure of Leeds General Infirmary's paediatric heart unit. Parents and children were faced with all the additional stress, risks and costs of travelling further for operations, and for others the horrendous unnecessary guilt as they wondered if their child’s outcome would have been better at another unit. There could not be a stronger case for professionals and decision makers using the risk adjustment model and communicating it well.”
Each hospital that performs children’s heart surgery in the UK and Ireland has had its overall survival rates published by the National Congenital Heart Disease Audit (NCHDA) since 2013. The researchers used PRAiS to calculate a predicted range of survival for each specific hospital, taking into account the complexity of each individual child’s medical condition and surgery. No hospital will have exactly the same predicted range of survival as another hospital, because each hospital treated different children.
However, the report the NCHDA publishes is lengthy, hard to find and hard to understand without expert knowledge. We ran user-testing workshops to involve the public, patients’ families and medical charities in co-designing the website with Pagel and Spiegelhalter, a first in this area.
“There is an understandable urge to put hospitals in a league table when comparing survival rates,” said Spiegelhalter, Winton Professor for the Public Understanding of Risk at Cambridge. “But rather than try and directly compare hospitals with each other, we need to compare a hospital’s survival rate with what we would predict it to be, taking into account how severe their cases are. This is a tricky idea but, with the help of many families, I think we have made it clear.”
“Because different hospitals treat different children and some children can have more complex medical problems than others, it is not valid to directly compare survival rates between hospitals,” said Pagel, Reader in Operational Research at UCL, who helped develop the formula the NHS uses to evaluate hospital survival rates. “We involved families from the beginning of the project and throughout to help us researchers communicate these complicated concepts clearly. I definitely learned that incorporating their feedback was absolutely crucial to building something useful. An accountable NHS is one where we can all understand how it is doing- and for this you need to listen to patients and families.”
Spiegelhalter and Pagel are calling for other researchers, companies and government to make health statistics accessible to patients and families by making them understandable. Transparency without accessibility is not enough; improved understanding by decision makers, health care professionals, patients and families can prevent misuse, confusion and unfounded anxiety.
Dr Rakow said: "Collaboration has been key to the success of this project: working with the mathematicians and statisticians who generate these kind of risk models, and using what we know from psychology about people’s understanding of statistical concepts, the team has thought carefully about what needs to be presented, and how.
"Then, having a highly skilled web developer and experienced science communications team from Sense About Science to refine and implement this – and with regular input from cardiologists, from the families of children with heart disease and the charities that support them – I believe we have developed something that should allow people to engage with what would otherwise be some fairly opaque information."
This project has also been written about in a variety of outlets:
- The BMJ/News: “Children’s heart surgery website aims to end confusion over survival rates”
- The Lancet/Editorial: “Communicating risk about children’s heart surgery well”
- Sense about Science wrote about why involving the patients and families, and involving them early, matters, on the Association of Medical Research Charities’s blog.
- Christina shared her insights from the experience and provides tips for other researchers in a National Institute for Health Research (NIHR) blog.
- The Royal Statistical Society blogged about the project.
- Tim Rakow, KCL shared his experience in a KCL news piece.
- University of Cambridge “Parent-led tool opens up NHS children's heart surgery data to families”
- Children’s Heart Federation explain: “ …parents’ views have been at the heart of developments that today launch a parent-led tool opening up NHS children’s heart surgery data to families.”
- Royal College of Surgeons of Edinburgh: “Understanding Children’s Heart Surgery Outcomes”
- Insight: Medical Research Council “Beyond open: making your data accessible”
- Understanding Uncertainty: “Exploring the language of chance in a sensitive context” by David Spiegelhalter