🔍 Read the full analysis: Assessing Research Ideas: The Role Of LLMs In Novelty Judgments on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Ai2 researchers report that six large language models produced unstable judgments of scientific idea novelty across evaluation setups. In one pairwise test, changing a few prompt words shifted accuracy by more than 50 percentage points; the team released a benchmark, data, code and an evaluation pipeline, while flagging limits in its labels.
Ai2 researchers report that six large language models gave unstable judgments about whether scientific ideas were novel, with a few prompt-word changes shifting accuracy by more than 50 percentage points in one pairwise experiment. The study challenges the reliability of using a single LLM score to screen research proposals, and Ai2 has released a benchmark, data, code and an evaluation pipeline for further testing.
The team tested GPT-5.1, GPT-5.2 and GPT-5.4, along with Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. The evaluations asked models to rate novelty directly and to compare ideas in pairs. The authors say results could change substantially across evaluation choices, including prompt wording.
The researchers also tested whether retrieving literature or allowing models more reasoning would make judgments more dependable. They report that neither intervention improved results consistently. Two purpose-built novelty systems also performed worse than a simple prompted Claude Opus 4.6 baseline, according to the study; this comparison does not establish that the baseline is reliable across other tasks or settings.
For a benchmark of human-written ideas, the team selected 154 ideas from accepted ICLR 2026 papers in the top-rated 10% of their research areas, where reviewers praised originality and no reviewer disputed it. A lower-novelty group included 145 ideas from rejected papers in the bottom 10%, selected when reviewers raised originality concerns and none praised novelty. In a check of 50 extracted reviewer signals, an annotator agreed with the direction in 47 cases; the authors say the three disagreements did not change the review classifications.
Why Unstable Scores Affect Research Screening
Research groups may use an LLM to rank candidate ideas before committing time to literature searches, experiments or expert review. If a judgment shifts sharply with small changes in the prompt or evaluation format, a score may reflect those choices as well as the idea being assessed. That makes comparisons across labs, projects or model versions harder to interpret.
The findings also bear on efforts to automate parts of scientific discovery. Systems that generate hypotheses or summarize prior work need a sound way to judge whether a proposal adds something new. The study does not show that every LLM novelty assessment fails. It does show why a strong result on one selected setup is not, on its own, evidence of dependable performance.
For readers evaluating AI research tools, the practical issue is whether the judgment remains stable when reasonable evaluation choices change. Ai2’s released resources offer other teams a way to test that question rather than rely only on a model’s score in a single configuration.
Products Worth Considering
Sony WH-1000XM6 Wireless Noise-Canceling Over-Ear Headphones, Blue, Bundle
Bose QuietComfort True Wireless Earbuds with Active Noise Cancellation, Black
Samsung 27-Inch Odyssey QD-OLED G8 4K 240Hz Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How Ai2 Constructed Its Novelty Tests
A novelty judge is a model asked to assess whether a research idea is new relative to prior work. It may judge an idea on its own or compare it with another proposal. The study authors say there is no standard evaluation recipe: researchers can vary the model, prompt, comparison format, reasoning budget and access to literature.
Ai2’s benchmark uses two settings. One relies on reviewer comments about human-authored ideas; the other pairs those ideas with proposals generated by a basic language model. In the latter setting, the researchers treated generated ideas as less novel, but describe those labels as weak rather than individually verified. In a separate expert check of 30 pairs—selected heavily from cases the strongest judge got wrong—the expert preferred the human idea in every pair and judged none of the generated ideas novel.
The authors caution that acceptance or rejection alone is not a reliable novelty label, because papers can succeed or fail for reasons other than originality. Their selection criteria instead draw on reviewer comments alongside paper ratings. Those signals support comparisons within the benchmark, but are not a definitive measurement of an idea’s originality.
““Seemingly small changes to the evaluation setup produce dramatically different judgments of the same ideas.””
— Ai2 study authors
Limits of the Benchmark Evidence
The benchmark’s novelty labels rely on reviewer comments and selection rules, not a definitive ground truth for originality. In the human-and-generated comparison, the generated ideas were assumed to be less novel; the authors acknowledge that this label was not checked idea by idea. The separate expert review covered 30 pairs, not the full dataset.
The supplied study description does not provide complete results for every model, prompt, reasoning budget or evaluation format, and it does not specify the release date. It also leaves open whether the reported instability carries over to other disciplines, idea generators or systems with broader literature access. The study does not establish which combination of model design, retrieval and prompts, if any, would produce consistent judgments in practical research use.
Further Tests Across Fields and Models
Ai2 has released the paper, benchmark data, code and an automated pipeline intended to help researchers create additional evaluation sets. Other teams can use those materials to check whether prompt sensitivity and disagreement between setups persist with different fields, sources of ideas and judge designs.
The supplied release information does not name a follow-up study or timeline, and it does not announce a method that resolves the reliability problem. The next evidence will need to show whether alternative evaluations can make judgments more consistent and whether that consistency holds beyond the benchmark’s reviewer-based labels.
Key Questions
Which models did the study test?
The researchers tested GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6, using direct novelty ratings and pairwise comparisons.
What did the prompt experiment find?
In one pairwise experiment, changing a few words in the prompt shifted accuracy by more than 50 percentage points. The supplied description does not give the full results for every prompt or model.
Did literature retrieval or more reasoning improve the results?
The authors report that retrieval and extra reasoning did not consistently improve the models’ novelty judgments. That does not show they cannot help in any particular setting.
Are the benchmark labels definitive?
No. The human-idea labels draw on reviewer comments and selection criteria, while the generated-idea group was treated as less novel without individual verification of every idea. The authors describe limits in both forms of evidence.
What has Ai2 made available?
Ai2 released the study, benchmark data, code and an automated pipeline for building additional evaluation sets. The supplied material does not give a specific date for the release or a schedule for follow-up work.
Primary source: Hugging Face · via ThorstenMeyerAI.com
