Authors: Adithya Venkatadri Hulagadri, Marzieh Fadaee, Jian Gang Ngui, Julia Kreutzer

When 17-year-old Gukesh Dommaraju became the youngest Chess World Champion in 2024 with his 2787 Elo rating, that number perfectly captured his competitive standing: stable, interpretable, and predictive of future match outcomes. Yet, when applied to ranking large language models, this same elegant system reveals fundamental limitations. Platforms like LMSYS's Chatbot Arena have adopted Elo to transform thousands of "A vs. B" chatbot comparisons into clean leaderboard rankings, but recent research from the Elo Uncovered paper demonstrates how Elo-based ratings suffer from volatility, order dependence, and violations of transitivity that undermine their reliability for AI evaluation.

The core mismatch lies in the nature of competition. Chess operates within strict, objective rules where outcomes are binary and player skill is consistent across contexts. LLM evaluation, by contrast, deals with subjective human preferences that are often ambiguous, culturally dependent, and multidimensional.

At the same time, traditional leaderboards that average performance across tasks — like SEA-HELM's original framework — obscure critical nuances in model capabilities. The core challenge lies in the nature of language itself. Unlike chess's binary outcomes, LLM evaluation deals with subjective, multidimensional judgments where task averages flatten important strengths (like exceptional translation capability) into aggregates that are difficult to interpret and Elo systems struggle with circular preferences and gaming vulnerabilities. Both approaches often fail to answer the most practical question: "Which model is more likely to give the better response?"

At Cohere Labs and AI Singapore, we've identified a third path forward, combining offline balancing with pairwise comparison systems like Bradley-Terry to produce more stable and interpretable rankings. By ensuring that diverse languages and tasks are equally represented, and by using a probabilistic framework that addresses Elo’s limitations, we move closer to evaluations that reflect real-world model utility.

In this post, we’ll introduce our offline pseudo-pairwise approach, share findings from the Elo Uncovered paper, and highlight insights from SEA-HELM, a multilingual evaluation suite spanning chat, instruction following, cultural understanding, and more. We close with concrete recommendations for building leaderboards that are transparent, fair, and better aligned with how language models are actually used.

Fundamental challenges of Elo for LLM evaluation

The transitivity problem in multidimensional systems

Elo ratings implicitly assume transitive relationships: if model A consistently beats model B, and B beats C, then A should outperform C. This mostly holds true in chess, where skill level is measured by how often a chess player wins, but fails dramatically for LLMs whose capabilities are tested across a vast array of domains — as if they were playing more than one type of game, like chess, Catan, basketball, etc. For instance, if a coding-specialized model could defeat a generalist at programming tasks while losing to the same model at creative writing, it creates circular dependencies that Elo cannot represent. These violations compound when evaluations span multiple languages or competencies, as seen in SEA-HELM’s framework where models might excel at Filipino cultural understanding while underperforming on Vietnamese reasoning tasks.

This multidimensionality directly impacts ranking stability. Traditional Elo systems compute updates incrementally after each matchup, causing ratings to fluctuate based on the sequence of battles — a phenomenon quantified in the Elo Uncovered study through controlled resampling experiments. When the same set of model comparisons were evaluated in different orders, the resulting Elo rankings showed significant divergence, with some models' positions varying by over 15 percentiles depending purely on battle sequencing.

The tie handling paradox

Ties are an unavoidable aspect of LLM evaluations as human judges frequently declare "both good" or "both bad" results, especially for ambiguous prompts or when comparing similarly capable models. Standard Elo implementations handle ties either by splitting game scores equally or by ignoring them entirely. The choice of approach may systematically overestimate skill gaps and underrepresent some tasks, or make it harder to distinguish between closely ranked models. Carefully accounting for ties is thus important to maintain both fairness and statistical reliability in model comparisons.

Figure 1: Elo scores of models in SEA-HELM with ties vs without ties. The range of Elo scores is reduced with ties around the pivot point of 1000, which is the arbitrary initial score assigned to new models.

In the case of multi-competency evaluations, some competencies may have higher propensity to lead to ties. Ignoring ties in the Elo computation would thus mean that these competencies are less represented in the final Elo score. In SEA-HELM, for example, one of our core objectives is to represent a holistic snapshot of models across languages and tasks through automatic methods. Elo scores can be helpful even for such a hierarchical leaderboard as they solve some of the aggregation problems we discussed in a previous blog post — namely that not every task’s score is linearly mappable to a 0-100 range, making averages hard to interpret.

In our experiments, we first calculated the Elo scores based on head-to-head contests within each competency of SEA-HELM: linguistic diagnostics, instruction-following, safety, chat, culture, natural language reasoning, natural language understanding, and natural language generation. Modeling the SEA-HELM Elo score as being made up of constituent competency Elo scores, we then performed linear regression through the origin (i.e., without an intercept) to estimate their linear relationship. We found that when ties are included, the coefficients of the respective competency Elo scores are much higher for competencies that have more granular metrics, as seen in the table below.

To illustrate the above point, for natural language generation, which includes tasks like translation and summarization, scores are calculated using a COMET-like regression model and ROUGE-L respectively. Thus, for a given sample, it is not easy for a head-to-head contest between models to result in a tie. In contrast, cultural evaluation uses multiple-choice questions, which can lead to ties more frequently. This issue can be exacerbated by these MCQ tasks sometimes being guessable, leading to more models arriving at the correct answer, which again results in ties.

As seen in the figure below, incorporating ties stabilizes each competency’s contribution to the final Elo score. It is thus important to maintain a proportionate representation of different types of tasks when incorporating ties to avoid exaggerating the model's strengths and weaknesses.

Figure 2: Relative contributions of Competency Elo scores to the SEA-HELM Elo score, with ties (in blue) and without ties (in green).

Gaming and selection biases of online leaderboards

Open evaluation platforms like Chatbot Arena, while valuable for crowdsourced benchmarking, are susceptible to strategic manipulation that can distort model rankings. One notable tactic involves redundancies in the prompt space: repeatedly submitting identical or near-identical prompts that exploit a specific model’s weaknesses. For example, if a model struggles with mathematical reasoning, flooding the arena with math-heavy prompts can artificially depress its Elo score as cumulative losses compound its rating decline. This practice mirrors "prompt spamming" in adversarial attacks, where targeted inputs skew perceived performance.

Recent studies, such as the am-ELO framework, propose mitigating these issues by replacing iterative updates with maximum likelihood estimation (MLE) and incorporating annotator reliability metrics, reducing score inconsistency by 30% compared to traditional Elo methods. However, even these improvements cannot fully eliminate risks like private testing loopholes, where vendors cherry-pick high-performing model variants before public release, skewing leaderboard standings.

Our leaderboard illusion paper shows that the very setup of these arenas can introduce selection bias particularly through non-uniform sampling of model variants, where developers disproportionately test and submit only the highest-performing iterations of a given model while suppressing weaker variants. This cherry-picking behavior artificially inflates the apparent capability of the model family as the arena leaderboard reflects a curated subset rather than the true distribution of performance. Empirical evidence reveals that certain organizations submitted over 20x more variants of flagship models for private testing compared to public release candidates, effectively gaming the Elo system through selective exposure.

Further compounding these challenges are arena-specific biases, such as human annotators favoring verbose or structured responses regardless of factual accuracy, which models can exploit to inflate their scores. The Elo system’s foundational assumptions — transitivity and reliability — are often violated in LLM evaluations, as demonstrated by studies showing significant volatility in rankings when the same battles are reordered.

**Figure 3**: Impact of the number of private variants tested on the best expected Arena score.

The figure above shows that as the number of privately tested variants grows, and the model provider gathers more Arena scores for these variants, the likelihood of selecting higher-performing models from the distribution also increases. This allows the provider to efficiently pinpoint the top-scoring model through iterative testing. However, if different providers have varying opportunities for private testing, the resulting rankings may become skewed.

Opponent selection matters

When evaluating language models using Elo scores, the choice of opponent matchups plays a crucial role in determining how quickly and accurately the system converges to a model's true skill level. Much like in chess, pairing models against balanced opponents — rather than consistently favoring stronger or weaker adversaries — leads to more reliable and faster convergence. If models are frequently matched against significantly stronger opponents, their Elo ratings may drop artificially due to high expected losses, while overly weak opponents inflate scores without meaningful challenge.

A well-designed evaluation system should prioritize balanced matchups, ensuring that wins and losses carry meaningful information about relative performance. This approach minimizes noise in the rankings and helps the Elo system stabilize around a model's true competitive strength more efficiently. Ultimately, thoughtful opponent selection is key to producing fair and interpretable LLM leaderboards.

**Figure 4**: Elo scores converge more rapidly to true skill levels when opponents are sampled uniformly from the distribution of model strengths.

Under biased sampling, where a new model initially faces predominantly stronger opponents, the model suffers repeated losses, causing its Elo score to underestimate its true skill. This results in slower convergence, as the model requires many additional matches to recover from the initial downward bias.

Solutions for robust LLM rankings

A statistically rigorous alternative: Offline pseudo-pairwise comparisons using annotated datasets

As discussed in our previous blog post, naive aggregation of automatic scores can obscure key insights, and SEA-HELM illustrates this well. It averages task scores within competencies, then across competencies within a language, and finally across all SEA languages. While this avoids over-weighting languages or tasks with broader coverage, it still flattens important differences. Score ranges vary across tasks, and outliers can skew averages, distorting overall model performance.

To address this, we recommend a complementary metric that asks a different question: “Is one model more likely to outperform another on average?” SEA-HELM’s per-sample outputs enable pairwise comparisons between models. With M models and N samples, we generate O(M^2*N) matchups and uniformly sample 200,000 across languages and competencies for balance. Elo scores are then computed using the Bradley-Terry model, which treats outcomes as wins, losses, or ties and estimates scores via batch maximum likelihood rather than incremental updates. This yields more stable, order-independent rankings and naturally accounts for ties. Bootstrapping produces confidence intervals shown on our leaderboard.

Since Elo reflects comparative strength and SEA-HELM scores reflect absolute performance, their rankings may diverge by design. Together, they offer a more complete view of model capabilities.

Balanced evaluation design

Strategic re-sampling can mitigate biases in Elo scoring by ensuring balanced representation across languages, tasks, and competencies, as demonstrated by the SEA-HELM implementation of Elo rating. Unlike Chess algorithms, LLMs are being used for an ever-increasing range of use-cases and languages, which need to be accounted for in Elo metrics. Think of LLM competitions as a pentathlon, not a sprint in a single race. For structured leaderboards with predefined prompt categories, this involves explicitly weighting contributions from underrepresented domains (e.g., low-resource languages or niche tasks) to prevent dominant categories from skewing rankings.

In a live leaderboard, dynamic re-sampling could periodically audit battle distributions and adjust sampling rates to maintain fairness. However, open-ended arenas pose a challenge, as prompt diversity is uncontrolled. Here, post-hoc stratification — clustering battles by latent topics or skills — could help identify and correct imbalances, though this requires robust metadata tracking. Alternatively, arenas could enforce minimum contribution quotas per language or task type, though this may limit organic user interaction.

Transparency and complementary metrics

To improve interpretability, we recommend live leaderboards to provide transparent, statistically rigorous Elo reporting. This includes displaying Elo scores with confidence intervals to convey uncertainty alongside granular breakdowns by language, task, or competency (e.g., "GPT-4’s Elo in Swahili coding tasks"). Pairwise hypothesis tests (e.g., t-tests or Bayesian comparisons) between models would clarify whether score differences are statistically significant, helping users distinguish meaningful gaps from noise. Crucially, documentation should explicitly define Elo’s limitations, for example, its sensitivity to prompt distribution and pairwise comparison biases, and discourage overinterpretation of small score differences. For dynamic arenas, real-time visualization of battle sampling rates (e.g., "60% STEM prompts in the last 24h") could further contextualize scores.

The next era of model evaluation

Elo scores alone are insufficient; leaderboards should integrate additional metrics to provide broader context. Pairwise win rates (as used in Stanford’s HELM) offer a simpler, more intuitive performance measure, though they lack Elo’s dynamic weighting. Aggregated normalized scores (like SEA-HELM’s task-specific benchmarks) can provide a grounding reference for Elo rankings in terms of absolute performance, which can reveal whether a "high Elo" model actually excels at target skills or just optimizes for arena preferences. For arenas, adversarial robustness metrics (e.g., consistency across prompt variations) could further differentiate models.

By combining Elo with these measures, leaderboards can better distinguish true capability from leaderboard gaming, though no single metric is perfect — transparency about trade-offs remains key.

Get involved

If building better LLM evaluation systems resonates with you, our Catalyst Research Grants can help bring your ideas to life. These grants represent Cohere Labs' commitment to supporting academic partners, civic institutions, and impact-focused organizations working on safe and responsible AI. Grants provide free access to the Cohere API to power research and projects, from fairer leaderboard designs to multilingual evaluation frameworks like the one in this blog.

This work is also part of our broader Open Science initiative. We’re not just another research group—we’re a hybrid lab with a dedicated research staff and a thriving open science community. Researchers, engineers, linguists, social scientists, and lifelong learners from over 100 countries connect here to collaborate on top-tier ML research, from small experiments to large-scale global projects.