The Protocol
PapersWithELO is an attention ranking protocol. It does not review papers and it does not predict acceptance. It orders a corpus so that human attention can be allocated where it matters — and so that the bottom of the ranking can be safely skipped.
1. Comparative, not absolute
LLMs are sycophantic graders: asked for an absolute score, they call everything “above average, 7/10”. Scores carry no ranking signal. So we never ask for scores.
Instead, a judge receives a batch of 6 papers (title + abstract + TL;DR) and must output exactly one thing: an ordering. Ranking is the only output we trust. Within a batch, relative differences are obvious even when absolute calibration is impossible.
2. Bayesian Plackett–Luce fitting
A batch ranking is a Plackett–Luce observation: the probability of an ordering over a set S factorizes as a sequence of choices,
We fit a latent quality θ per paper by maximum a posteriori estimation under a Gaussian prior, and approximate posterior uncertainty with a Laplace approximation at the mode. Every paper therefore carries θ ± σ — a distribution, not a score. Papers with few comparisons honestly report wide uncertainty.
3. Two judges, model-agnostic fusion
Each year is ranked independently by two judges — mimo-v2.5-pro and deepseek-v4-pro — in separate tournaments with separate θ fits. θ scales are judge-dependent, so we z-score within each judge before fusing:
No judge is load-bearing. A future judge can be added by running its tournament and re-fusing — old ratings stay immutable. This is what makes the protocol future-proof rather than a one-off experiment.
4. Incremental: the new-entry protocol
A protocol is only useful if new papers can join without recomputing the world. A newcomer is compared in a handful of batches against fixed incumbents whose ratings are frozen. The newcomer receives a conditional θ and percentile; existing ratings do not move.
ratings_latest is immutable. This is the same property that made PageRank a protocol rather than a ranking of the web as it existed in 1998.
5. Mass-submission adjustment
Submission flooding is a real incentive distortion. We compute author credibility from submission count and expose it as a transparent, toggleable percentile penalty — off by default, shown as an explicit per-paper delta, never hidden inside θ. Aggregate statistics only; no author leaderboards, no names.
6. Cost & reproducibility
| Year | Papers (10% sample) | Judge A (mimo) | Judge B (deepseek) |
|---|---|---|---|
| 2024 | 710 | $78.40 | $121.90 |
| 2025 | 1,140 | $132.70 | $208.30 |
| 2026 | 1,900 | $210.50 | $340.20 |
Sampling: numpy, random.seed(42), 10% of ICLR 2024/25/26. Ground-truth decisions from OpenReview. All ratings, observations, and judge identities are auditable offline; the site ships only aggregated JSON.
7. Future: cascade filtering & BYOK
Cascade filtering: cheap judges screen the full corpus; expensive judges re-rank only the boundary region (percentiles 20–60), where decisions actually flip. Expected ~5× cost reduction at equal boundary precision.
Bring-your-own-key: rank your own reading list (up to 10 arXiv papers) with the same batch protocol, directly from your browser against OpenRouter — your key never touches our servers. (In development.)
See it in action on the 2026 leaderboard or the validity analytics.