ChatGPT, the generative artificial intelligence (AI) that is said to have unlimited potential to be utilized in medicine, has a limitation because it needs to be validated by a doctor at the final decision-making stage before it can be applied in the clinical field, experts said. 

Professor Kim Seok-hwi at the Department of Pathology at Ajou University School of Medicine, recently published an article, titled, “In the Era of ChatGPT, Can Medical Artificial Intelligence Replace the Doctor?,” in the Korean Journal of Medicine. 

"It's been a long time since the medical community started asking the question of whether AI will replace doctors, but before ChatGPT, most of the responses were somewhat cynical," Kim said.

"Recently, however, the atmosphere has changed a lot, and doctors seem to both admire and be threatened by recent advances in AI models."

To consider whether an AI model, such as ChatGPT, can be applied in clinical practice, Kim said that doctors need to look at whether it can solve unmet needs and whether it can replace them. 

Based on this, he predicted that it would be possible to let an AI model help doctors make decisions that are difficult to perform in reality.

If the unmet demand is too high, and if doctors can't perform, AI models can be used clinically, Kim went on to say.

"For example, counting whole blood cells in blood tests is a necessary process for major clinical decisions, but if the machine can count accurately and the function is verified, doctors will trust it and make decisions,” he said.

In any area, if the results can be checked by a physician and a final judgment made, these models can be seen as extending the role of the physician and the scope of medical care rather than replacing it, Kim noted.

However, he added, AI is unlikely to be applied to tasks that doctors can do without difficulty. In a structure where doctors are responsible for all medical decisions, if AI takes over, it could lead to unclear responsibilities, he said.

"For this reason, even when developing AI models, the ability for doctors to review the final decision is a critical gateway," Kim said.

ChatGPT will be able to enter the medical field only if doctors are the ones who finally check and decide on the output of the model.”

However, it will be difficult to apply an AI model based on publicly available data, such as ChatGPT, to the clinical field because it cannot provide experts’ final verification of the information, he said.

For a research model to be applied in the medical field, it must go through a process that includes setting up a research hypothesis and plan, an institutional review board’s review, annotation of ground-truth and model development, validation, proving superiority or non-inferiority for specific indications in the clinical field, and applying to regulatory agencies for licensing the results of the clinical trial.

"Since the basic premise of the ChatGPT model is an unrestricted public dataset, there is no device to supervise the appropriateness of the data," Kim said.

"The contents of the latest papers that need to be paid for access can be excluded, and unreliable articles posted on general blogs may be included as key data. The lack of expert decision-making in the model development process is also a risk."

The medical community will not let go of evidence-based medicine, and only models developed by a group of experts with a sophisticated design will be able to reach real patients, he said. 

"Any AI model, including ChatGPT, must be validated and finalized by doctors before it can be adopted in the real world."