Cohere Labs - Tom Jacobs, PhD student at the CISPA Helmholtz Inst.

Classical statistics teaches us that overparameterization causes overfitting, which prevents good generalization. However, highly overparameterized neural network architectures generalize surprisingly well. This is because the training of these models tends towards low rank or sparse solutions, without requiring explicit constraints. This preference is known as implicit regularization, and it can be found in a variety of contexts, including attention layers, LoRA, matrix sensing, and diagonal linear networks. As a result, implicit regularization helps explain how overfitting is avoided and generalization is improved in neural networks.

In this talk, I will show how weight decay controls implicit regularization beyond its explicit role of constraining the model capacity. For instance, it moves the implicit regularizer from $L_2$ to $L_1$, which leads to more sparsity in the model. This demonstrates how weight decay not only serves as a model constraint, but also has an implicit effect. By turning off weight decay during training, only the implicit effect remains, resulting in better generalization overall. Besides better generalization, I use these insights to induce sparsity in deep neural networks. Sparsity aims to reduce model size and inference time by removing as many weights as possible. This results in a new method: PILoT (Parameteric Implicit Lottery Ticket), a sparsification method based on overparameterization and weight decay that uses the transition of the implicit regularization from $L_2$ to $L_1$ to gradually sparsify, achieving high sparsity with a smaller performance drop.

Tom Jacobs is a PhD student at the CISPA Helmholtz Institute in Saarbrucken, Germany, under supervision of Rebekka Burkholz. He currently works on understanding training dynamics in deep learning and designing methods for efficiency. Before that he has graduated with a master in applied mathematics specializing in mathematical analysis and probability theory from the Technical University of Eindhoven, in the Netherlands.

Add event to calendar

Apple Google Office 365 Outlook Outlook.com Yahoo