Was curious how LLMs such as Chat GPT and Grok 3 were interpreting medical imaging scans such as MRI, CT, X-ray, etc.. so I asked them directly 😀
It seems Chat GPT is using :
Visual Feature Analysis: (LLM-based reasoning, not CNNs)
1. Pattern recognition: Based on training with many labeled examples (text-image pairs)
2. Medical knowledge: Reasoning using anatomical, pathological, and radiological concepts.
They are not using convolutional neural networks (CNNs) like some established radiology AI tools.
This is really interesting because they are learning by associating images of scans with associated descriptive text, I assume from digitized books or reference articles when the LLM is trained. This is similar to how basic book knowledge is obtained by radiology residents. They LLM's should eventually get better and learn to diagnose textbook medical cases consistently.
In clinical practice, majority of the cases are not "classic" textbook cases, whether you have to infer a novel concept or diagnose with artifact/suboptimal images. This is where practical knowledge is gained in residency and with experience. It is definitely possible the LLMs can make this leap as well...very interesting times.