ChatGPT tested in giving correct answers on cardiac arrest and cardiopulmonary resuscitation

Cardiac arrest raises a series of questions among witnesses of the event, survivors and their families. When it involves a young person or a public figure, the attention on issues such as cardiac arrest and cardiopulmonary resuscitation (CPR) increases significantly. The Internet is often used as a tool to find information on medical issues. More recently, ChatGPT, an artificial intelligence model, offers the ability for anyone to get human answers on a variety of topics, including health and medicine.

However, considering the risks associated with access to inaccurate information in the medical field, the team of physicians and researchers led by Dr. Tommaso Scquizzato, resident in Anesthesia and Resuscitation, and Prof. Giovanni Landoni, UniSR Professor of Anesthesiology and director of the Center for Research in Anesthesia and Intensive Care at IRCCS Ospedale San Raffaele, evaluated the effectiveness of ChatGPT in answering questions from those who are not experts in the field of cardiac arrest and cardiopulmonary resuscitation (CPR).

Cardiac arrest survivors and their family members were involved to conduct the evaluation to obtain direct feedback on the responses provided by ChatGPT. This allowed incorporating the perspective of those who may use these systems in the future. The research results were published in the journal Resuscitation.

ChatGPT, introduced by OpenAI in November 2022, is an advanced natural language processing tool based on artificial intelligence (AI) technologies. It allows users to conduct human-like conversations and provides detailed answers on a wide range of topics. ChatGPT knowledge comes from a diverse source of information available on the Internet, including books, articles, websites, and other publicly accessible written materials.

“Although most of the contents that the artificial intelligence draws on are reliable, it is worth highlighting that ChatGPT could also learn incorrect or partially inaccurate information from the network or incorrectly reprocess what it has learned. If this happens with questions relating to health problems, the consequences could also be serious. For example, in the case of cardiac arrest, ChatGPT could provide incorrect indications on how to intervene or provide inadequate explanations to the family members of a person suffering from cardiac arrest. For this reason we decided to test and evaluate ChatGPT's ability to answer a series of questions that survivors and their families typically ask us on a daily basis"

says Tommaso Scquizzato, first author of the study.

Artificial intelligence (AI) has the potential to bring enormous benefits in healthcare. However, it also raises ethical issues that must be carefully considered to ensure responsible use aligned with societal values. Find out about them in the in-depth look at Hi-tech Medicine, the exhibition/workshop created by UniSR in collaboration with Microsoft Italia and Porini on artificial intelligence in the service of health.

The researchers co-produced a list of 40 questions together with cardiac arrest survivors and their family members. The answers provided by ChatGPT (March 2023 version) to each question were evaluated by 14 professionals (physicians, nurses and psychologists) for their accuracy, by professionals and lay people (cardiac arrest survivors and their family members) for their accuracy. their relevance, clarity, completeness and overall value on a scale of 1 (poor) to 5 (excellent) and for readability.

“We asked these questions to ChatGPT and the answers received received an overall positive rating with the exception of the answers relating to cardiopulmonary resuscitation which were judged to be the worst in terms of correctness"

explains Dr. Marco Mion, clinical psychologist at the Essex Cardiothoracic Center in Basildon in the United Kingdom and last author of the study.

“A concrete example of a ChatGPT error was noted when we asked it to explain how to perform cardiopulmonary resuscitation on a woman. ChatGPT advised the user to place their hands in the center of their chest, between their nipples. The correct indication that ChatGPT should have given – clarifies Dr. Arianna Gazzato, nurse of the Cardio-Thoraco-Vascular Intensive Care Unit – is to compress the center of the chest with the hands one on top of the other, placed on the lower half of the sternum" .

Generative AI models like ChatGPT will play a significant role in the future, so medical content generated by these systems should be monitored and moderated.

For further details on cardiopulmonary resuscitation, consult the Italian Resuscitation Council website

"This study – concludes Professor Giovanni Landoni – is the result of a fruitful collaboration between our Anesthesia and resuscitation research center at the IRCCS San Raffaele Hospital, the Essex Cardiothoracic Center in Basildon, the Anglia Ruskin School of Medicine, Chelmsford, United Kingdom and Sudden Cardiac Arrest UK, a British association that brings together cardiac arrest survivors and their families. The active inclusion of patients and their families in the design and execution of the study represents a distinctive and innovative element of this project that allowed us to conduct a more in-depth and comprehensive analysis".

The initiative is part of the Public Engagement activities to which the San Raffaele Campus has been passionately and enthusiastically dedicated for years, with a view to promoting a culture of health and prevention in society and the territory through opportunities for involvement and mutual learning between researchers and the public.