Researchers rely on ML models created by companies to conduct research. One such model, OpenAI’s Codex, has been used in about a hundred academic papers1. Codex, like other OpenAI models, is not open source, so users rely on OpenAI for accessing the model. On Monday, OpenAI announced that it would discontinue support for Codex by Thursday. Hundreds of academic papers would no longer be reproducible: independent researchers would not be able to assess their validity and build on their results. And developers building applications using OpenAI’s models wouldn’t be able to ensure their applications continue working as expected.OpenAI asked users to switch to GPT 3.5 with less than a week’s notice. Source.The importance of reproducibility Reproducibility—the ability to independently verify research findings—is a cornerstone of research. Scientific research already suffers from a reproducibility crisis, including in fields that use ML. Since small changes in a model can result in significant downstream effects, a prerequisite for reproducible research is access to the exact model used in an experiment. If a researcher fails to reproduce a paper’s results when using a newer model, there’s no way to know if it is because of differences between the models or flaws in the original paper. OpenAI responded to the criticism by saying they’ll allow researchers access to Codex. But the application process is opaque: researchers need to fill out a form, and the company decides who gets approved. It is not clear who counts as a researcher, how long they need to wait, or how many people will…OpenAI’s policies hinder reproducible research on language models