News
OpenAI blasts the AI evaluation "benchmark": Nearly 30% of the 731 questions are defective, and the pass rate has soared from 23% to 80% in 8 months. OpenAI published a blog post to publicly challenge the industry's authoritative evaluation benchmark SWE-Bench Pro, believing that about 30% of the 731 public test tasks have evaluation defects.
3 min read
Source: Telegram AI频道
OpenAI blasts the AI evaluation "benchmark": Nearly 30% of the 731 questions are defective, and the pass rate has soared from 23% to 80% in 8 months and has failed. OpenAI published a blog post to publicly challenge the industry's authoritative evaluation benchmark SWE-Bench Pro, believing that about 30% of the 731 public test tasks have evaluation defects. SWE-Bench Pro was launched by Scale AI, specifically to evaluate the programming capabilities of large language models and AI agents. Because it is highly close to actual enterprise-level development and has extremely high anti-cheating standards, it has become the industry's authoritative benchmark in the field of AI software engineering. OpenAI pointed out a key signal in the blog post: the passing rate of the cutting-edge model on this benchmark soared from 23.3% to 80.3% in just 8 months. The speed of this "progress" is too abnormal. OpenAI believes that this benchmark can no longer effectively evaluate the model's real software development capabilities. It is probably because there are systemic problems in the evaluation itself, rather than because the model's capabilities have truly achieved such a leap. The two review paths were cross-verified, and nearly 30% of the tasks were "unqualified". To verify the judgment, OpenAI launched two parallel review paths. Its data point analysis process marked 200 failed tasks, accounting for 27.4% of the total 731 public tasks; the manual labeling activity carried out at the same time identified 249 failed tasks, accounting for 34.1%. Based on the cross-validation of these two paths, OpenAI estimates that about 30% of SWE-Bench Pro tasks have defects, involving four types of problems: too strict testing, insufficient prompts, too narrow a test scope, and misleading prompts. OpenAI also disclosed a typical case: a certain question requires adding 1 space at the beginning of the line when converting the content to Markdown, but the hidden test requires 2 spaces. This means that the model will still be judged wrong if the code is written exactly according to the instructions in the question. This kind of problem of "inconsistency between hidden requirements and explicit instructions" directly leads to the misevaluation of the true capabilities of the model, and also explains why the pass rate has soared unreasonably. Withdrawal of Adoption Suggestion and Calling for the Reconstruction of the AI Evaluation System Based on this analysis, OpenAI officially withdrew its previous adoption recommendation for SWE-Bench Pro. OpenAI believes that in the future, senior software developers will need to design new benchmarks specifically for AI evaluation, rather than simply using the testing logic for human developers. When the industry "benchmark" itself may have nearly 30% defects, the credibility of the entire AI evaluation system will be questioned. Returning from running score competition to real engineering capability evaluation may be the next threshold that must be crossed in AI software engineering evaluation. via AI News (author: AI Base