Chat GPT outperforms skilled medical doctors and has a go at diagnosing the Black Death

On Nov 18, 2024

Chat GPT is better at diagnosing conditions than doctors are, an alarming study has found. Even when doctors used Chat GPT, the chatbot alone still outperformed trained medics.

Dr Adam Rodman, who helped designed the study, was “shocked” by the findings. The expert in internal medicine at Beth Israel Deaconess Medical Center in Boston expected chatbots would help doctors diagnose illnesses. However, the doctors in the study who used ChatGPT-4 along standard resources, performed only slightly better than the medics who didn’t use the chatbot.

The chatbot and doctors were given six case studies of hard to diagnose and rare conditions. The OpenAI tech scored an average of 90% when diagnosing a medial condition from a case report and explaining its reasoning, whereas doctors using the bot scored 76%. The docs not using the tech at all scored 74%.

Mother nursing sick child At Home — Chat GPT was found to be better at diagnosing difficult conditions than actual doctors
(Image: Getty Images)

Not only did the study highlight that the chatbot was superior to trained doctors, it revealed that doctors can hold a resolute belief in their diagnosis, even if a chatbot suggests a better one.

With such concerning results, the Daily Star wanted to test the study’s findings and Chat GPT’s ability, with our own, scientifically-stringent experiment.

Two flagellants flogging themselves during plague epidemic in Europe — Determining that my armpit boils could be the bubonic plague was child’s play for the bot
(Image: Getty Images)

The bot may be all over cholesterol embolism and back pain, but what does it know about medieval conditions like the bubonic plague or leprosy? To test this, I typed the following into the bot: “I have the following symptoms, boils under armpits, groin and neck, sudden fever, headache, chills. What illness do I have?”

Chat GPT was quick to reply with “I’m not a doctor”, before listing five potential ailments, with the Black Death as the top diagnosis.

Doctor Checking Skin of Sick Girl — But it stumbled on diagnosing my leprosy
(Image: Getty Images)

It also proffered up that I could be suffering from boils, lymphadenitis (infected lymph nodes), a type of staph infection or “other bacterial or viral infections”.

Next, I tried to catch the bot out with leprosy. I asked: “I have lost my eyebrows, my nose has changed shape, my toes have shrunk and I have a stuffy nose and sound like a pug when breathing. What illness do I have?”

Chat GPT told me that my symptoms, including my distinctive “pug-like” breathing sound were “quite unusual” and could suggest a serious underlying condition, such as “rheumatoid arthritis”, the thyroid disorder “Graves’ disease” or “chronic sinusitis”, but no leprosy.

Doctor in Venice at the time of the plague — Missing eyebrows, shrunken toes and pug-like breathing led the bot to rheumatoid arthritis
(Image: De Agostini via Getty Images)

I tweaked my symptoms to lost eyebrows, misshapen nose, shrunken toes, a chronically stuffy nose and numb skin lesions that won’t heal, and chat GPT came up with autoimmune disease Scleroderma.

It seems like I had really flummoxed AI with leprosy, but I thought I’d give it one last try, and told the bot my symptoms were numb skin lesions that won’t heal, a loss of sensation, crusting of the nose and that “I’m bleeding from my eyes”.

Eyam village church grave yard skull and cross bones carving on the grave stone from the bubonic plague era — Yet a bleeding eyeballs and a crusty-rimmed nose, could be leprosy, the bot found
(Image: Getty Images/iStockphoto)

The bleeding eyeballs and crusty nose combination did the trick, and on a list of seven utterly grim health conditions, Leprosy popped up in third position.

Lesson learned; if you are bleeding from your eyeballs, consult your laptop immediately.

For the latest breaking news and stories from across the globe from the Daily Star, sign up for our newsletters.