Mayo physicians find gaps but also promise in Google’s medical AI model Med-PaLM 2 | | | | Turtles AI

  Researchers at Mayo Clinic have been testing Google's latest medical AI model Med-PaLM 2 since May. The testing focused on evaluating the model's capabilities for clinically relevant tasks. Med-PaLM 2 is based on PaLM 2, a next-generation language model that boasts huge improvements in multilingual processing, logic reasoning and code generation. Med-PaLM 2 is specifically focused on medical operations, such as interpreting X-rays similar. Quoting from Google site, "it can answer questions and summarize insights from a variety of dense medical texts. It achieves state-of-the-art results in medical competency, and was the first large language model to perform at “expert” level on U.S. Medical Licensing Exam-style questions. We're now adding multimodal capabilities to synthesize information like X-rays and mammograms to one day improve patient outcomes. Med-PaLM 2 will open up to a small group of Cloud customers for feedback later this summer to identify safe, helpful use cases".   Med-PaLM 2 still suffers from some of the accuracy issues we’re already used to seeing in large language models. In the study, physicians found more inaccuracies and irrelevant information in answers provided by Google’s Med-PaLM and Med-PalM 2 than those of other doctors. Still, in almost every other metric, such as showing evidence of reasoning, consensus-supported answers, or showing no sign of incorrect comprehension, Med-PaLM 2 performed more or less as well as the actual doctors. The physicians identified several areas where Med-PaLM 2 was helpful, including providing summaries of research papers, explaining complex medical concepts, and guiding diagnosis decisions. Researchers concluded that while not perfect, Med-PaLM 2 shows promise as a decision support tool for doctors - especially for tasks requiring the review of diverse information sources quickly. The study recommends Google focus on improving Med-PaLM 2's factual accuracy and ability to highlight relevant details while minimizing false or extraneous content. With further refinements based on physician feedback, medical AI models like Med-PaLM 2 may eventually be able to augment doctors' knowledge and reduce cognitive loads.