Multilingual LLMs are rapidly improving, but their generative evaluation lacks rigor and consistency. Drawing from machine translation practices, our work proposes best practices and a checklist to improve evaluation quality and enable better development of mLLMs.

Kaleidoscope is a new benchmark for evaluating vision-language models across 18 languages and 14 subjects, with 20,911 multiple-choice questions. It addresses the lack of multilingual and multicultural coverage in existing benchmarks.

We present a multi-faceted evaluation framework that measures not only performance but also fairness, unintended effects, and adaptability across varying levels of preference divergence.

Large Language Models (LLMs) are increasingly used in working environments for a wide range of tasks, excelling at solving individual problems in isolation. However, are they also able to effectively collaborate over long-term interactions?

This policy primer outlines some of the challenges around measuring AI model efficiency systematically, and the techniques being developed to improve model efficiency. It focuses on work that can be done at the model developer layer, as opposed to the hardware or energy supply layers.

In this research, we explore the possibility of achieving greater fairness by using an imbalanced dataset instead of a balanced one, and how the use of ensembles could further amplify this impact. We carry out the same analysis on real datasets, such as CheXpert and CelebA.

Progress in AI is driven largely by the scale and quality of training data. Despite this, there is a deficit of empirical analysis examining the attributes of well-established datasets beyond text. In this work we conduct the largest and first-of-its-kind longitudinal audit across modalities.

We introduce the Aya Expanse model family, a new generation of 8B and 32B parameter multilingual language models, aiming to address the critical challenge of developing highly performant multilingual models that match or surpass the capabilities of monolingual models.

This Policy Primer summarises several promising avenues to addressing the language gap in AI safety and identifies five recommendations for researchers and policymakers to consider in their efforts to improve AI safety for everyone.