Cohere Labs - Sara Papi, AI Researcher at Fondazione Bruno Kessler

Users expect modern multimodal LLMs (MLLMs) to operate over spoken and video inputs in multiple languages, flexibly addressing diverse task requests such as transcription, translation, summarization, and question answering. Despite the central role of this instruction-following ability, existing evaluations often remain limited to text-only and monolingual settings, failing to reflect the complexity of real-world scenarios. As a solution, this talk describes the crosslingual evaluation of general-purpose instruction-following models (including SpeechLLMs, VideoLLMs, and MLLMs) through MCIF, a novel benchmark built from scientific talks. Developed entirely through human annotation, MCIF has been designed to assess how the instruction-following abilities vary across languages, modalities, and tasks. Through extensive benchmarking of 23 state-of-the-art models, MCIF reveals critical performance gaps that monolingual single-modality benchmarks cannot capture, such as significant limitations in joint speech-video integration, long-form processing, and summarization, establishing a foundation for advancing truly multimodal, crosslingual instruction-following systems.

Sara Papi is an AI Researcher at Fondazione Bruno Kessler (FBK), Italy, where she works on speech processing and multimodal LLMs within the MEETWEEN and DVPS Horizon European projects. She received her PhD cum laude in Information Engineering and Computer Science from the University of Trento in 2024, with a focus on simultaneous speech translation and subtitling. Her research interests span multimodal and crosslingual instruction-following models, speech foundation models, and LLMs. Her work has been recognized with awards, including the Best PhD Graduate 2024 Award in Information and Communication Technology from the University of Trento, an Outstanding Paper and SAC Award at ACL 2024, and a Social Impact Paper Award at EMNLP 2024. She actively contributes to the community as an organizer of the IWSLT Evaluation Campaign and as an Area Chair or reviewer for major conferences in speech and NLP, such as *ACL and Interspeech.

Add event to calendar

Apple Google Office 365 Outlook Outlook.com Yahoo