Cohere For AI - Rohan Jain, MSc Student in ML at U of Calgary

The Lottery Ticket Hypothesis (LTH) suggests that there exists a sparse winning ticket mask and weights that achieves the same generalization performance as the dense model while using significantly fewer parameters. LTH achieves this by iteratively sparsifying and re-training within the pruned solution basin. However, this procedure is computationally expensive, and a winning ticket’ssparsity mask does not generalize to other weight initializations. Recent work has suggested that Deep Neural Networks (DNNs) trained from random initialization find solutions within the same basin modulo weight symmetry, and proposed a method to align trained models within the same basins. We propose permuting the winning ticket mask to align with the new optimization basin when performing sparse training from a different random initialization. Using this permuted mask, we show it is possible to significantly increase the generalization performance of sparse training from random initialization with the permuted mask as compared to sparse training naively using the non-permuted mask. We empirically demonstrate that our proposed method improves the generalization of LTH with the new random initialization on multiple datasets (CIFAR-10/100 and ImageNet) using VGG11 and ResNet-20/50 models of varying widths.


Bio: "My name is Rohan. I'm a 2nd year MSc student at the University of Calgary in Canada working under Dr. Yani Ioannou. My research interests lie in understanding the geometry of NN loss landscapes, sparse neural network training dynamics, mean-field theory, and random matrix theory. I studied Mathematics & Computer Science during my undergrad at the University of Toronto in Canada, with a focus in probability & combinatorics."

Add event to calendar

Apple Google Office 365 Outlook Outlook.com Yahoo