Use of large language models in clinical research (part 2)

The involvement of technological giants such as Microsoft and Google in the development of large language models (LLMs) for general public makes this technology likely to be soon omnipresent, since any activity on a computer highly depends on their products.

Despite the current global craze, LLMs are not a new technology. Its concepts were developed more than half a century ago. LLMs are, as described by ChatGPT itself, when prompted to provide a definition, “a type of artificial intelligence that have been designed to process and generate human-like text. These models are trained on vast amounts of data, such as books, articles, and websites, and use this information to learn about language and how it is used. As a result, large language models are able to generate coherent text that is difficult to distinguish from text written by a human”.

ChatGPT is based on an LLM called GPT-3.5 (GPT – Generative Pre-training Transformer), a neural network machine-learning model that uses the same pre-training datasets as its precursor GPT-3, which was released in 2020, with additional fine-tuning. This fine-tuning stage includes the incorporation of a concept called “reinforcement learning with human feedback“ (RLHF). ChatGPT is based on GPT-3.5, but works within even stricter guardrails on ethical standards, by introducing human values via strict ethical policies and rules to the model. It is assumed the list of these rules and policies was probably similar to the one published by DeepMind’s Sparrow, which was designed by DeepMind researchers, California Institute of Technology, University of Toronto, and University College Dublin (1, 2). Finally, ChatGPT was GPT optimized for dialogue (3).

GPT-4 is a large multimodal model, which accepts not only text, but also image inputs and delivers text outputs. It is more capable than its predecessor GPR-3.5 in terms of reliability, creativity and handling specific instructions. However, the limitations reported by OpenAI, seem to be similar to those of GPT-3.5, therefore, human review is still needed. (4)

In the previous blog post of this series, we have reported already on the first experiences with ChatGPT for medical writing tasks. In this blog post we focus on the significant limitations and pitfalls that need to be taken into account when using LLMs.

Despite some very impressive skills and interesting results that are delivered quickly, there has been a general consensus among the users, especially within the research and medical community, that current LLMs cannot be considered as a reliable and trustworthy tool yet. The users have been reporting how some results are plain false, are purely made up by the AI tool, and provided references are incomplete. Deceptively, texts are so coherently and confidently written that they can be easily mistaken as a text created by a competent human being (5). We need to remind ourselves, however, that the output has been created by a non-thinking tool. It does not grasp the meaning of the results, and, more importantly, it cannot verify if the provided information is correct. “Contents are a reflection of a collective internet; its creations are generic averages” as put in one of the opinion pieces (6, 7). We also have to keep in mind that the models we explored were developed for a more general purpose and therefore not fine-tuned for medical writing.

Several factors of utmost importance need to be taken into consideration before this technology can be reliably used in everyday practice:

It is known that the LLMs are trained on a big amount of data. In case of ChatGPT the knowledge it received consisted of open source databases until 2021. One can ask the model to take also a specific reference in consideration and this way prompt it into writing more reliable results. The models can be trained by using only a specific set of data sources and references, excluding unwanted or unethical content, as defined by the trainer (i.e., the company that owns the model). Thereby, LLMs can offer more reliable results, whereas, on the other hand, they could be manipulated into a specific direction (knowingly and unknowingly), and therefore provide biased, outright false, even dangerous results. There have also been concerning reports on the ethical implications on the human input needed in the selection of appropriate results (8).

At this moment, accuracy of the results is questionable. Very prominently, upon the announcement of Google Bard’s release a blatant mistake was spotted in the given text (and the company suffered immediate damage on the stock market). Without a laborious factual check, the texts cannot by any means be directly used (9). Also, for some LLMs the data sources are not verifiable, meaning they may contain incorrect information. The incorrect information is therefore being disguised as a correct information, and may be disseminated to the audience by an unsuspecting user. OpenAI itself acknowledges this problem as a limitation on the first page of ChatGPT:

The Limitations that show up on the first page of ChatGPT. Screen shot, source: https://chat.openai.com

An example of a medical background section on headache as written by ChatGPT. Legend: input from GKM medical writer,  ChatGPT output. Screen shots have been merged for better readability. Source: https://chat.openai.com