Cohere Labs - Ambroise Odonnat PhD Student

Large language models (LLMs) are remarkably efficient across a wide range of natural language processing tasks and well beyond them. However, a comprehensive theoretical analysis of the LLMs' generalization capabilities remains elusive. In our paper, we approach this task by drawing an equivalence between autoregressive transformer-based language models and Markov chains defined on a finite state space. This allows us to study the multi-step inference mechanism of LLMs from first principles. We relate the obtained results to the pathological behavior observed with LLMs with high temperatures, such as repetitions and incoherent replies. Finally, we leverage the proposed formalization to derive pre-training and in-context learning generalization bounds for LLMs under realistic data and model assumptions. Experiments with the most recent Llama and Gemma herds of models show that our theory correctly captures their behavior in practice.

Ambroise Odonnat is a PhD student at Huawei Noah's Ark Lab and Inria, advised by Romain Tavenard, Laetitia Chapel, and Ievgen Redko. His goal is to improve the core understanding of Transformers by conducting theoretical analysis and large-scale experiments on large language models, transformer fine-tuning, and out-of-distribution generalization.

Add event to calendar

Apple Google Office 365 Outlook Outlook.com Yahoo