Building off the success of Aya, in July, Cohere For AI launched Expedition Aya: a six-week, global open-build challenge to connect researchers worldwide and support new multilingual research initiatives. Over 176 researchers signed up, working across 28 teams on a diversity of projects to push the boundaries of multilingual AI. Since the Expedition concluded at the end of August, many teams have continued to collaborate, after showcasing their progress at a public, virtual competition, judged by Sebastian Ruder, Marzieh Fadaee, and Alice Schoenauer Sebag. We’re proud to celebrate the five projects that were honored with the top recognition.

Gold - MLCC: Multilingual Climate Chatbot

Captained by Luis Ticas and Helena Yu, with collaboration from team members Maryam Tavakoli Kashi, Avneet Kaur, and Karan Verma.

A multilingual chatbot system for improving community data literacy and increasing their participation in climate adaptation.

Judges’ note: The MLCC: Multilingual Climate Chatbot team shared a very complete and thorough treatment of an impactful and timely topic. We were impressed by what they were able to accomplish in the short timeframe. They designed RAG and chatbot architectures with thought-out workflows that trigger for climate-related questions. They conducted extensive evaluations including both automatic and human evaluation. On top of this, they demonstrated how users would use their project via a demo, and shared a clear timeline of the next steps. We're really excited for the launch of this project and to see how the MLCC team will develop it even further.

Silver - Evaluating Reward Models in Multilingual Settings

Captained by Srishti Gureja, with collaboration from team members LJ Miranda, Shayekh Bin Islam, Drishti Sharma, Rishabh Maheshwary, and Gusti Winata.

Studying the ability of open-source reward models to evaluate multilingual preferences with a comprehensive analysis and creation of the first multilingual reward dataset.

Judges’ note: The Evaluating Reward Models in Multilingual Settings team tackled an important and understudied topic. We think the dataset they created through translation is going to be quite impactful and useful to the community. Beyond that, the evaluation and analysis conducted by this team will already provide immediate value and insights. We appreciated that they evaluated a range of models and clearly presented key takeaways and observations that were already quite interesting, including the impact of using different word choices across different languages. Finally, they even created another language-mixed reward modeling dataset that should be useful to the community.

Bronze - Sanskriti Aya: Designing Linguistic Quality Index to Analyze Conversational Capabilities of LLMs

Captained by Guneet Singh, with collaboration from team members Aniruddha Walke, and Saloni Shah.

Developing a linguistic quality index framework that evaluates and enhances LLM conversations across languages.

Judges’ note: The Sanskriti Aya project is an impressive and difficult work that builds an important bridge between NLP and linguistics. We appreciated how much thought they put into the linguistically motivated evaluation framework that they developed. The analysis in English and different Indian languages was extensive and nuanced. We are excited to see this type of project and want to encourage more cross-disciplinary work like this. Their in-depth analysis helps us understand how LLMs model language on a more fine-grained linguistic level. We hope to see more work from this group as they further develop their framework and results.

Most Innovative - The Language Effect

Captained by Christopher Klamm, with collaboration from team members Avneet Kaur, Ali Abdelkader, Ashay Srivastava, Vishnu Lanka, Arnav Arora, Rosemary Igwe, and Abhipsha Das.

Studying what role the interaction language plays in retrieving political knowledge and opinions in multilingual models?

Judges’ note: We were very intrigued by the questions from the Language Effect team, that is, what the impact of language (English, versus Hindi, German etc.) is on political bias in LLM generations. We were impressed by all they have done to study these questions: building a solid multilingual dataset, looking at behavioral inter-language agreement, and doing a thorough analysis of toxicity, all in this political domain. The findings were also really interesting. We are very excited about the innovation behind framing this question in a fresh way. The outcome of this project will be very useful for a lot of people and can hopefully inform how to diminish this bias for the next generation of models.

Most Promising - Maya: Multimodal Aya

Captained by Nahid Alam, Karthik Reddy Kanjula, Surya Guthikonda, with collaboration from team members Satyajit V, Anthony Susevski, Roshan Santhosh, Drishti Sharma, Iftekhar Uddin, Sangyeon Kim, Shayekh Bin Islam, Timothy Chung, Snehanshu, Snegha, Ashvanth.S, Abhipsha Das, Ryan Chan, Isha Chaturvedi, Kamya Hari, Genta Indra Winata, Alham Fikri Aji, Cecilia, Sai Vivek Mulukuri, Shreyas Patil, and Reem Abdel-Salam.

Bridging the gap in multilingual vision-language models.

Judges’ note: The Maya: Multimodal Aya project is very, very timely. Lately, it seems there have been multimodal models coming out every day, but almost none of them cover many languages. There is a strong need for setting a high standard in this domain, and for adding multimodal capabilities to multilingual leaderboards. We were very impressed by what this team has achieved in six weeks, and are very excited to see what they will do next with this project.

Congratulations to all Expedition Aya teams on their dedication to pushing the boundaries of multilingual AI. We are excited to follow along with your journeys and continue to support work, which is helping to connect people worldwide. See all Expedition Aya teams in the recording of our Closing Ceremony.

Empowering Exploration of the Next Frontier - Expedition Aya