Today, Cohere For AI, Cohere’s research arm, is proud to announce Aya Expanse, a family of highly performant multilingual models that excels across 23 languages and outperforms other leading open-weights models.

We are releasing Aya Expanse as both 8 and 32 billion open-weights models available on Kaggle and Hugging Face, as part of our continued commitment to multilingual research and to accelerate the frontier for multilingual AI. The 8 billion parameters model makes breakthroughs more accessible to researchers worldwide, and our 32 billion parameters model offers state-of-the-art multilingual capabilities.

Aya Expanse marks an important step to expand high-quality coverage of languages in LLMs. Since we first launched the Aya initiative two years ago, we have collaborated with over 3,000 researchers from 119 countries to expand cutting-edge multilingual research. This included releasing the Aya collection, the largest multilingual dataset collection to-date, with 513 million examples, and critical evaluation sets for multilingual performance and safety. It has also included the release of Aya-101, the most comprehensive multilingual model to-date covering 101 languages.

A Spotlight on Multilingual Research

At a larger scale, Aya Expanse 32B outperforms Gemma 2 27B, Mistral 8x22B, and Llama 3.1 70B, a model more than 2x its size, setting a new state-of-the-art for multilingual performance. We also released Aya Expanse 8B, which outperforms the leading open-weights models in its parameter class such as Gemma 2 9B, Llama 3.1 8B, and the recently released Ministral 8B with win rates ranging from 60.4% to 70.6%.

The improvements in Aya Expanse are the result of a sustained focus on expanding how AI serves languages around the world by rethinking the core building blocks of machine learning breakthroughs. Our research agenda for the last few years has included a dedicated focus on bridging the language gap, with several breakthroughs that were critical to the current recipe: data arbitrage, preference training for general performance and safety, and finally model merging. Below, we spotlight this research.

Advancing synthetic data for languages with limited data. The use of synthetic data – data generated by an expert or “teacher” model to train another model – has become increasingly central to the development of LLMs, particularly as model training has exhausted data sources. However, for multilingual data, especially with languages that are low-resource, there are few good examples of teacher models, creating an extra challenge to leveraging synthetic data.

We proposed a novel data sampling strategy that we term data arbitrage to avoid mode collapse, or the generation of “gibberish” when over relying on synthetic data. Data arbitrage takes inspiration from how humans learn by going to different teachers for different skills. If you want to learn to play piano, you’d go to a piano teacher. To learn to bake, you'd seek out a baking expert. We apply the same philosophy to AI, strategically selecting different “teacher” models based upon the data distribution to generate suitable synthetic data for multilingual capabilities.

Guiding models towards global preferences. Preference training is used in the late stages of model training, leveraging feedback from humans to guide the model toward what high-quality outputs look like. We think of it as the "final sparkle” in training an AI model. However, preference training and safety measures often overfit to harms prevalent in Western-centric datasets. Problematically, these safety protocols frequently fail to extend to multilingual settings. Our work is one of the first that extends preference training to a massively multilingual setting, accounting for different cultural and linguistic perspectives. We find this leads to large gains both in general performance and safety.

Model merging in a diverse world. The final step to our breakthrough in multilingual performance is our work on model merging – combining the weights of multiple candidate models at each stage to create more versatility and performance.

Putting together our research to build the best multilingual model in its class. We combined all of these techniques in one training recipe for Aya Expanse. Each of these techniques – from data arbitrage to merging and multilingual preference optimization – enable step-by-step improvement, leading to a significant gain against the other leading models in the same parameter classes. Against Gemma 2 9B, Aya Expanse 8B achieves 60.4% win rate, representing a new frontier for its class in multilingual LLM performance.

Scaling To The State-of-the-Art in Multilingual.

Each algorithmic breakthrough we developed over the year and unified in Aya Expanse, scales with the larger model size and enables state-of-the-art performance. Leveraging this potential, we used the same training recipe combining data arbitrage, preference training, and merging. Similar to our smaller model, this recipe leads to a significant improvement in performance. Aya Expanse 32B outperforms Gemma 2 27B, Mistral 8x22B, and the much larger Llama 3.1 70B with pair-wise win rates ranging from 51.8% to 76.6% overall in 23 languages.

These results showcase the impact of our research on multilingual, beyond small language models, marking a new state-of-the-art in multilingual performance at scale.

Get Involved

In addition to our release of open weights for 8b and 35b, we are actively collaborating with the wider research ecosystem to accelerate multilingual AI. Apply for one of our research grants here or join our next research cohort where we provide compute resources and expertise to promote cross-institutional collaboration.

Aya is part of our continued commitment to greater multilingual performance and safety. Find out more about our research here, and consider joining us.

Aya Expanse models are available today for researchers and developers on the Cohere API platform, Kaggle, and Hugging Face.

Acknowledgments

This work is the result of a huge amount of effort across both the Cohere For AI and Cohere teams. A huge thanks also goes to our research community – the 220 language ambassadors from around the world who have been part of this release. Listed in alphabetical order below:

Mehmet Emre Akbulut (Turkish/Turkey), Samer Attrah (Dutch, Arabic/The Netherlands), MUHDIN AWOL (English/Ethiopia), Kenza Benkirane (Arabic, French/Morocco), Mann Bhanushali (Hindi, English/India), Isabella Bicalho Frazeto (French, Portuguese/Brazil, France), Danylo Boiko (Ukrainian/Ukraine), Sabrina Boumaiza (Arabic/Algeria), Samuel Cahyawijaya (Indonesia/Indonesia), Samuel Cahyawijaya (Indonesian, English/Indonesia), Emirhan Çelik (Turkish/Turkiye), Ryan Chan (Italian, English/United Kingdom), Aurélien-Morgan CLAUDON (French/France), Urszula Czerwinska (Polish, French/France, Poland), Joana da Matta (Portuguese/Brazil), Nguyễn Đạt (Vietnamese/Vietnam), Manasvi Dawane (Hindi/India), Akanksha Devkar (Hindi/India), Sharad Duwal (English/Nepal), Abdeljalil EL MAJJODI (Arabic /Morocco ), Shafagh Fadaei (Persian/Iran), Neil Fernandes (German, Japanese, Hindi/Canada), Silvia Fernandez (Spanish, English/Argentina), Hamidreza Ghader (Persian, Dutch/Iran, Netherlands), Manuel Goulão (Portuguese/Portugal), Bassam Gouti (Arabic, Tunisian dialect/Tunisia), María Grandury (Spanish/Spain), Miguel Guerrero (Spanish/España), Siddhesh Gunjal (Hindi/India), Mohammed Hamdy (Arabic, English/Egypt), Hafedh Hichri (Arabic, French/Tunisia), Nhu Hoang Anh Quynh (Japanese, Vietnamese/Japanese, Vietnamese), Kyle Howard (English, Spanish/United States), Jiwung Hyun (Korean/South Korea), Joseph Marvin Imperial (Filipino/Philippines), Burin Intachuen (Japanese/Japan), Ryan Junejo (English, Hindi/Canada), Juan Junqueras (Spanish/Argentina), Karthik Reddy Kanjula (English, Hindi/USA, India), Albert Kao (Chinese /Taiwan), Morteza Kashani (Persian/Iran), Ahmed Khaled (Arabic/Egypt), Niharika Khanna (Hindi, Punjabi/India), Dipika Khullar (Hindi/United States, India), Christopher Klamm (German/Germany), Nazar Kohut (Ukrainian/Ukraine), Alkis Koudounas (Italian, Greek/Italy, Greece), Diana Kozachek (German, Ukrainian/Germany), Katrina Lawrence (English, French, German /Canada), James León (Spanish /Ecuador ), Jiazheng Li (Chinese/China, UK), Nicolò Loddo (Italian/Italy), Dante, Fu On Lok (Chinese, English/Hong Kong), Iro Malta (Greek, German/Greece, Germany), Harras Mansoor (Urdu/Pakistan), Bhavnick Minhas (Hindi/India), Shachar Mirkin (Hebrew), Roa’a Mohammad (Arabic/Jordan), Yiyang Nan (Chinese/China), Sree Harsha Nelaturu (Hindi, German/India, Germany), Jekaterina Novikova (English, Russian/Canada), Roni Obaid (Arabic, English/Syria), Olympiah Otieno (English/Kenya), Enes Özgözler (Turkish/Turkey), Yavuz Alp Sencer Ozturk (Turkish/Turkey), Carlos Patiño (Spanish/Colombia), Jebish Purbey (English, Hindi/Nepal), Maria Quijano Jesurum (Spanish/Colombia), Swati Rajwal (Hindi, English/USA), Didi Ramsaran Chin (Spanish, English/Venezuela), Divyaraj Rana (Hindi, English/United States), Aditya Retnanto (Indonesian/Indonesia), Rodrigo Ribeiro Gomes (Portuguese/Brazil), Esra'a Saleh (English, Arabic/Canada), Roshan Santhosh (Hindi/India, USA), Drishti Sharma (Hindi, Spanish/India, Spain, Latin American countries ), Kinza Sheikh (Urdu/Pakistan), Aditya Shrivastava (Hindi, English/India, Ireland), Vivek Silimkhan (Hindi, English/India), Marjana Skenduli (English, Italian/Albania), Soham Sonar (Hindi, Marathi/India), Gürkan Soykan (Turkish/Türkiye), David Styveen (Portuguese/Brazil), Anthony Susevski (English/Canada), Adrian Szymczak (Polish/Poland), Joanne Tan (English, Mandarin Chinese/Singapore), Quentin Tardif (French/France), Ameed Taylor (English, German/USA, Germany), Yiorgos Tsalikidis (Greek/Greece), Roman Tymtsiv (Ukrainian/Ukraine), Muhammad Saad Uddin (German/Germany), Louis Ulmer (French/France), Sundar Sripada V. S. (English/United States), Freddie Vargus (English, Spanish/United States), Vlad Vasilescu (Romanian/Romania), Karan Verma (English Hindi/India), Henry Vo (Vietnamese/Vietnam), Minh Chien Vu (Japanese, Vietnamese/Japanese, Vietnamese), Hieu Vu (Vietnamese/Vietnam), Azmine Toushik Wasi (English, Hindi, Arabic, Spanish/Bangladesh), Warren Williams (English/United States), Joseph Wilson (French/Canada), Gusti Winata (Indonesian/Indonesia), Ege Yakut (Turkish, English/Türkiye), Eray Yapağcı (Turkish/Turkey), Taha Yassine (Arabic, French/Morocco ), Serhan YILMAZ (Turkish/Turkey), Hanna Yukhymenko (Ukrainian/Ukraine), Mike Zhang (Dutch/Netherlands).

Aya Expanse: Connecting our world