Cohere Labs - Tony Silveti-Falls, Assoc. Professor at CentraleSupélec

Tony will discuss optimization methods that leverage the linear minimization oracle (LMO) over a norm-ball and their application to training huge neural networks. In recent work, Tony has proposed with his coauthors a new stochastic family of algorithms that uses the LMO to adapt to the geometry of the problem. The resulting update rule unifies several existing optimization methods under a single framework, including spectral methods like Muon, which we are able to prove rigorous convergence results for. Furthermore, we propose an explicit choice of norm for deep architectures, which, as a side benefit, guarantees the transferability of hyperparameters like learning rate across model sizes. Experimentally, we demonstrate significant speedups on nanoGPT training using our algorithm, Scion, without any reliance on Adam. The proposed method is memory-efficient, requiring only one set of model weights and one set of gradients, which can be stored in half-precision.

Antonio (Tony) Silveti-Falls is an associate professor of artificial intelligence at CentraleSupélec in the department of mathematics and a member of the Inria team OPIS. He works at the intersection of {stochastic, nonsmooth, noneuclidean} optimization theory and machine learning, with a focus on developing and analyzing algorithms for large-scale problems.

Add event to calendar

Apple Google Office 365 Outlook Outlook.com Yahoo