Aug 18, 2026
The IOL-AI Challenge: An Open Challenge towards Advancing Linguistic Reasoning
We hosted an open competition on unseen linguistic reasoning problems to measure and advance AI reasoning skills. The official IOL 2026 jury graded AI solutions, and one frontier model scored equivalent to a gold medal.
Authors
Eduardo Sánchez*, Rita Berrada*, Dan-Mircea Mirea*, Sara Rajaee, Alexander Piperski, Ana Meta Dolinar, Boris Iomdin, Andrey Nikulin, Mariya Shmatova, Marzieh Fadaee, and Julia Kreutzer *First authors
Abstract
Reasoning in LLMs is overwhelmingly studied in domains that provide a model with rules: mathematics and code. Linguistic puzzles invert this: the solver must first discover the system before reasoning within it. We present the IOL-AI Challenge, an open-science competition run on the unseen problems of the International Linguistics Olympiad (IOL) 2026 Individual Contest, evaluated both automatically and, for the first time, by members of the official IOL Jury under the same rubrics applied to human contestants. The challenge drew 731 submissions from 46 teams under a strict compute budget (one T4, 30 mins). We additionally benchmark 15 unconstrained frontier and open models, with Claude Opus 4.8 earning a jury score equivalent to a gold medal, while both resource-constrained systems we submitted for jury grading scored in the range of the bottom 5\% of contestants. Capability was not determined by scale: 14B submissions outperform models twice their size, and gains come from decoding and output-handling rather than model capacity. We also found that automatic metrics rank systems exactly as the jury does, but compress the scale, upscoring weak systems by ${\sim}13$ points and understating strong ones.
Related works
Research
The Art of Asking: Multilingual Prompt Optimization for Synthetic Data
Read
Research
Unlocking Reasoning Capability on Machine Translation in Large Language Models
Read
Research
Findings of the WMT25 Multilingual Instruction Shared Task: Persistent Hurdles in Reasoning, Generation, and Evaluation
Read