ChatGPT-4, an artificial intelligence chatbot that has the ability to carry human-like conversation, has been in the news with regular frequency over the past few months. Writers in Hollywood wanted restrictions on AI involvement in their latest collective bargaining agreement, professors are constantly worried that writing assignments will be outsourced to ChatGPT, and we’re interested to see how the technology will change the medical industry for the better (or worse).
Recently, researchers decided to put ChatGPT to the test to see how it compared to real doctors in diagnostic and decision-making capabilities for the identification and treatment of soft-tissue foot and ankle issues. So how did the AI learning bot stack up? Will you soon be turning to a chatbot to help with your latest ankle issue?
ChatGPT Diagnostic Study
For the study, researchers presented eight soft-tissue-related foot and ankle cases to ChatGPT, with each case assessed by three fellowship-trained foot and ankle orthopedic surgeons. The evaluation system included five criteria within a Likert scale, which assesses information on a 5 to 25 point scale, with five being the lowest and 25 being the highest possible score.
ChatGPT fared pretty well in its evaluation and diagnostic capabilities, as the average sum score of all eight cases was 22.0. Its assessment of Morton’s neuroma received the highest grade (24.7), while its assessment and care recommendations for the peroneal tendon tear received the lowest grade (16.3). Researchers went on to say that subgroup analyses of each of the five criterion used showed no notable differences compared to surgeon grading. That said, the chatbot was markedly better at diagnosing, providing treatment recommendations and providing accurate information than it was at providing alternate treatments or providing comprehensive information.
After assessing all of the results, researchers concluded that ChatGPT-4 could effectively diagnose and provide reliable treatment options for the soft-tissue foot and ankle cases that it was presented with. They did note some limitations with alternative treatment recommendations and a comprehensive depth of information, but overall the quality of the assessment and care recommendations was high. Researchers were also pleased to see that the chatbot did not fabricate treatment options or recommendations, as this had been seen in previous studies in different sectors. The team concluded that providers could point patients towards ChatGPT-developed patient education materials with relative confidence, although it may not be the best source for comprehensive information.
ChatGPT should continue to help with the diagnostic and treatment phases of medical care, but we don’t see it eliminating the need for real orthopedic surgeons anytime soon. Quick access to countless databases and previous cases can prove helpful, but every patient needs to be analyzed on an individual basis, so what has worked for others isn’t guaranteed to work for the latest patient. It’s certainly interesting, but it’s not coming for our jobs just yet!