en.wikipedia.org/wiki/PaLM
1 correction found
Med-PaLM was the first to obtain a passing score on U.S. medical licensing questions
The primary Med-PaLM literature says Flan-PaLM, not Med-PaLM, was the first model to clear the MedQA/USMLE-style passmark.
Full reasoning
This sentence appears to misattribute the milestone. In the original Med-PaLM paper, the authors say that Flan-PaLM "achieves ... 67.6% accuracy on MedQA (US Medical License Exam questions)" and then explain that Med-PaLM was introduced afterward as a new aligned model: "To resolve this we introduce instruction prompt tuning ... The resulting model, Med-PaLM..." In other words, the paper presents Flan-PaLM as the model that first crossed the benchmark threshold, with Med-PaLM coming later.
A later follow-up paper makes this even more explicit in its introduction: "Our initial model, Flan-PaLM, was the first to exceed the commonly quoted passmark on the MedQA dataset comprising questions in the style of the US Medical Licensing Exam (USMLE)." That directly contradicts the Wikipedia sentence crediting Med-PaLM as the first.
So the issue is not that Med-PaLM performed poorly; it is that the article assigns the first passing-score milestone to the wrong model in the PaLM medical-model lineage.
2 sources
- Large Language Models Encode Clinical Knowledge
Flan-PaLM achieves ... 67.6% accuracy on MedQA (US Medical License Exam questions) ... To resolve this we introduce instruction prompt tuning ... The resulting model, Med-PaLM...
- Towards Expert-Level Medical Question Answering with Large Language Models
Our initial model, Flan-PaLM, was the first to exceed the commonly quoted passmark on the MedQA dataset comprising questions in the style of the US Medical Licensing Exam (USMLE).