IdeaTaste
A benchmark for research taste
1Carnegie Mellon University 2Mohamed bin Zayed University of Artificial Intelligence
01 · Idea
Freeze the literature at the end of 2025 and let LLM agents read it and write research proposals. In 2026 the same community publishes. Taste is how much the two overlap, in what they cite and in what they propose.
Research taste, the judgment of what to read, which problems to pursue, and which methods to try, increasingly shapes the work of AI scientists, yet it remains far less understood than their ability to execute research. We introduce IdeaTaste, a benchmark that scores an LLM by how closely the research proposals it writes from a literature snapshot frozen at 2025 match the work the same communities published in 2026. Our experiments span six contemporary language models and four research areas from established to emerging fields, covering more than 90,000 proposals in total.
LLM agents fall well short of human communities in both the literature they cite and the ideas they state, reaching about 26% of the human level on citation matching and only 18% on idea matching. Their populations can cite as diversely as a community and take up its tasks and methods, yet rarely arrive at the concrete approaches and research plans it pursues. Neither richer field context nor larger populations substantially close this gap. In blinded expert review, AI proposals trail the 2026 papers least in presentation and most in expected success, echoing their gap in concrete approaches, and those whose ideas no human paper shares fare worst, suggesting weaker directions rather than overlooked opportunities. These results indicate that the main bottleneck in AI research taste lies less in finding the directions a community works on than in turning them into the concrete approaches it goes on to pursue.
Writing research means two choices: which prior work to build on, and what idea to state in relation to it. IdeaTaste compares LLM agents with a human community on both, using what the community actually did after the cutoff as the reference.
For each area, papers published through 2025 form the corpus U that LLM agents search and cite. Papers published January–August 2026 form the human reference H. Both sets grow by following citations out from 20–22 hand-picked landmark papers per area. The reference is screened against a written area definition, and surveys are removed.
| Area | Corpus U | Reference H |
|---|
Each document is described by its citation set (cited papers in U) and by an idea summary in five slots, extracted from its abstract: contribution type, problem, motivation, proposed approach and research plan.
Scoring the 2026 papers against one another gives the human level that every population is read against.
All methods share one ReAct agent with a 20-step budget: plansearchread_papersdraftsummaryproposal. It can cite only papers it has retrieved. The methods differ only in what the agent knows about its area.
Only told it is a researcher in the area. Reveals what it pursues without a prior.
Assigned one research topic, allocated in proportion to the corpus with recent years upweighted.
Sees the area's most common task tags, method tags and task–method pairs with their shares, and can retrieve papers by tag.
Topic, run in rounds. Later rounds can search and read earlier rounds' proposals and research traces.
Instead of asking a judge whether an idea sounds novel, IdeaTaste asks whether the community went on to pursue it. The 2026 papers are the reference, and scoring them against one another sets the human level.
Across 96 populations they reach about 26% of the human level on citation matching and 18% on idea matching, even though their citations are as spread out as a community's.
Populations take up the problems and motivations a community works on, yet almost never arrive at the concrete approaches and research plans it pursues.
Richer field context barely moves matching, and growing a population to the size of the human reference lifts it only gently. Unmatched proposals also fare worst in blinded expert review.
02 · Experiments
96 populations (6 models × 4 methods × 4 areas), 300 proposals each, scored against the 2026 papers.
Each population holds 300 proposals. Models are ordered by release date; the two released before 2026 cannot have seen any 2026 paper. Click a column to sort.
| Model | Method | MatchCite | MatchIdea | DiversityCite | DiversityIdea |
|---|
Averaged over all 96 populations, LLM agents reach about a quarter of the human level on citation matching and under a fifth on idea matching. Their citations are as spread out as a community's, often more so, but their ideas repeat one another.
Split by slot, populations match the problems and motivations of human papers many times more often than their approaches and research plans. Almost no human approach or plan finds a counterpart among the proposals. This holds for a model trained before 2026 (Llama 4 Scout) and for one released after (DeepSeek-V4-Flash).
03 · Analysis
Matching barely moves across methods, while diversity changes several-fold. Giving a direction (Topic, Frontier) spreads a population out; showing the full map of an area (Landscape) helps least of all.
Growing Free and Topic populations from 300 proposals to the size of the human reference (3.8–17× larger) lifts matching only gently. Even at equal size it stays well below the human level.
Eleven reviewers scored 192 papers and proposals without knowing their source. Proposals trail the 2026 papers by only 4 points on presentation but 20 on expected success, so the gap lies less in how they are written than in whether they would work. It is widest for proposals whose ideas no human paper shares.
For two 2026 papers, two proposals from one model share the problem. Only one of them also matches the approach.
Fetching uncached mixture-of-experts experts causes substantial I/O overhead.
Pretrained world models must adapt to changing environments while retaining prior capabilities.
Live · Taste Tracker
Every quarter we add new models and extend the human reference with newly published papers. Proposals written from the frozen snapshot stay on a watchlist: when the community later publishes the same idea, the proposal counts as verified.
Editions · the window of human papers each one is scored against
Score as a share of the human level (best method per model, averaged over areas), against model release date. Hollow points: released before the corpus cutoff.
Want to help? Review ideas blind and your scores join the human review set in a coming edition.
AI proposals written from the 2025 snapshot, and the later human paper that verified them (same problem and same approach). A first sample from the paper; the full watchlist arrives with edition 2027.01.
More cases coming soon.
2Mohamed bin Zayed University of Artificial Intelligence@article{zhong2026ideataste,
title = {IdeaTaste: Do LLMs Share the Research Taste of Human Researchers?},
author = {Zhong, Shanshan and Xu, Chen and Ning, Jingjie and Liu, Zhengzhong and Kumar, Aviral and Xiong, Chenyan},
year = {2026}
}