Happy to announce that our paper was rejected as a spotlight (5/5/4) at #ICML2026. If the methodology was complex enough to confuse the metareviewer, perhaps it may still be of broader interest to you 🙂. Happy to discuss the work if you are into optimal counterfactual maps that permit explanations in milliseconds, or into the occasional ups and downs of academic publishing 🚣

Qianqian and I were metareviewer pairs in last year WACV and she is an amazing person, besides being an amazing researcher.

The paper builds a practical way to plug AI into peer review without replacing humans. Finds a careful AI workflow can get near human accuracy while adding consistent, useful feedback at scale. An AI metareviewer reached 81.8% accept or reject accuracy, close to the 83.9% human average. The system, called ReviewerToo, reads a paper, checks the literature, collects multiple AI reviews, then writes a single verdict. It uses reviewer personas like empiricist, theorist, and pedagogical, and each follows the same conference rubric. An author agent drafts a rebuttal, and a metareviewer weighs the evidence, filters weak claims, and makes a recommendation. They tested this on 1,963 real submissions from a major machine learning conference. Single personas show bias, for example permissive leans accept and critical leans reject, so mixing them reduces that bias. Ensembles and the metareviewer match decisions best and give more actionable feedback than many single reviews. AI reviewers are strong at fact checking and literature coverage. AI reviewers are weaker on judging real novelty and deep theory. Rebuttals can make agents too positive, so the authors keep humans in charge of final calls. ---- Paper – arxiv. org/abs/2510.08867 Paper Title: "ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Review"

But that metareview is useless anyway. I hope the metareviewer is never invited back.

Well, I can relate, just got rejected with 4332 and metareviewer explicitly writing that he disregards 2-score due to the fact that we addressed the concerns. Also, metareviewer explicitly wrote recommendation to accept

if i remember correctly my accepted acl paper last round had scores of 1, 2, 2.5 post-rebuttal (metareviewer score 4) yes, 2/3 of those reviews were entirely worthless

Thrilled to announce that my paper on media background checks has been accepted to #EMNLP Findings! 🎉 Very happy to see this, especially because the metareviewer was an LLM - and not great. Their review helps to illustrate an argument from my paper, though! 🧵

We faced a similar turnaround by a metareviewer who didn't really get the crux of our deliberate setup, and took it for voluntary anomalies. Unfortunate.

Yeah I am not sure being silent helps. If a reviewer submits a factually incorrect review or ask clarification questions that are addressed in rebuttal not responding wastes everyone’s time and places too much burden on metareviewer

Very nice experience submitting to @COLM_conf The metareviewer did a great job taking both the reviews and the rebuttal into account Hopefully the same quality can be maintained as the conference grows bigger

I've been so lucky this year as a metareviewer for @ISMIRConf: 1 Reject, 3 Accepts, and 1 Strong Accept (to be recommended for best paper)! I usually reject the majority of my assigned ISMIR papers. Maybe I'm getting old? Or maybe I just want everyone to come to San Francisco 🌉

I dont think there is a standard way of dealing with this. For sure write a confidential comment to editors/metareviewer explaining situation. Maybe say something vague in rebuttal, eg "Thank you for highlighting XX, we will examine it and decide how to best refer to this work"

Our Learn page now includes over 15 relevant and high-useable blog posts! Are we missing something important? Let us know in the comments or send us an email to: metareviewer@air.org

Always nice when your sacrifices (of time) to the metareviewer gods bears fruit

Obligatory ICLR tweet: DSPy accepted as Spotlight. 1. This could be a notable start of a pattern at ML confs—as our metareviewer says, "refreshing to see this unconventional research style (developing programming models) at ICLR" 2. Time for a crisp thread on DSPy & what's new?

MetaReviewer has simple defaults for the novice, and complex flexibility for the methodologist! Find out what we mean by viewing our newest product: the MetaReviewer Explainer Video. youtu.be/s3ygZ_KttAk

Take a look at what #MetaReviewer has to offer in our #FullRelease on 11.01.23.

Want to see more features and new additions? Join us for the 🥳Full Release 🥳Launch Webinar on 11.01.23. air-org.zoom.us/webinar/regi… #metaanalysis #systematicreview #MetaReviewer #MetaReviewerMondays #evidencesynthesis

I've seen a few of my res referred to as "Kontorovich's inequality". It was rather amusing/gratifying to read, as a metareviewer, the reviewers argue among themselves whether the paper in question was applying K's ineq. correctly. Can't recall what result they were referring to!

The metareviewer quoted an isolated negative sentence in otherwise positive reviews ("no practical use", but this is no engineering paper) and criticized studying a multilingual setup instead of focusing on English (yeah, you read right. And reviews didn't mention that). (2/3)