It’s no secret that large language models (LLMs) like the ones that power popular chatbots like ChatGPT are surprisingly fallible. Even the most advanced ones still have a nagging tendency to contort the truth — and with an unnerving degree of confidence. And when it comes to medical data, those kinds of discrepancies become a whole lot more serious given that lives may be at stake. Researchers at New York University have found that if a mere 0.001 percent of the training data of a given LLM is “poisoned,” or deliberately planted with misinformation, the entire training set becomes likely to propagate errors. As detailed in a paper published in the journal Nature Medicine, first spotted by Ars Technica, the team also found that despite being error-prone, corrupted LLMs still perform just as well on “open-source benchmarks routinely used to evaluate medical LLMs” as their “corruption-free counterparts.” In other words, there are serious risks involved in making use of biomedical LLMs, which could easily be overlooked using conventional tests. “In view of current calls for improved data provenance and transparent LLM development,” the team writes in its paper, “we hope to raise awareness of emergent risks from LLMs trained indiscriminately on web-scraped data, particularly in healthcare where misinformation can potentially compromise patient safety.” In an experiment, the researchers intentionally injected “AI-generated medical misinformation” into a commonly used LLM training dataset known as “The Pile,” which contains “high-quality medical corpora such as PubMed.” The team generated a total of 150,000 medical articles within…If Even 0.001 Percent of an AI's Training Data Is Misinformation, the Whole Thing Becomes Compromised, Scientists Find