OpenAI researchers have admitted that even the most advanced AI models still are no match for human coders — even though CEO Sam Altman insists they will be able to beat “low-level” software engineers by the end of this year. In a new paper, the company’s researchers found that even frontier models, or the most advanced and boundary-pushing AI systems, “are still unable to solve the majority” of coding tasks. The researchers used a newly-developed benchmark called SWE-Lancer, built on more than 1,400 software engineering tasks from the freelancer site Upwork. Using the benchmark, OpenAI put three large language models (LLMs) — its own o1 reasoning model and flagship GPT-4o, as well as Anthropic’s Claude 3.5 Sonnet — to the test. Specifically, the new benchmark evaluated how well the LLMs performed with two types of tasks from Upwork: individual tasks, which involved resolving bugs and implementing fixes to them, or management tasks that saw the models trying to zoom out and make higher-level decisions. (The models weren’t allowed to access the internet, meaning they couldn’t just crib similar answers that’d been posted online.) The models took on tasks cumulatively worth hundreds of thousands of dollars on Upwork, but they were only able to fix surface-level software issues, while remaining unable to actually find bugs in larger projects or find their root causes. These shoddy and half-baked “solutions” are likely familiar to anyone who’s worked with AI — which is great at spitting out confident-sounding information that often falls apart on closer inspection. Though all…OpenAI Researchers Find That Even the Best AI Is "Unable To Solve the Majority" of Coding Problems