SILO: Characterizing structure in deep classifiers through Neural Collapse

In this talk I will describe recent progress in characterizing structure that emerges during the training of deep neural networks for classification. Neural Collapse is a phenomenon that emerges during the training of deep classifiers in which the top-layer feature embeddings of samples from the same class tend to concentrate around their means, and the top layer’s weights align with those features. We first show how it emerges when training deep networks with weight decay and normalization. We then investigate if these properties extend to intermediate layers. We use the basic tool of studying the within-class and between-class covariances to describe how deep classifiers perform their task. Finally we will explore the implications of Neural Collapse in explaining generalization in deep learning.

Akshay Rangamani is an Assistant professor of Data Science at the New Jersey Institute of Technology. Prior to this he was a postdoctoral associate at the Center for Brains, Minds, and Machines at MIT, and received a fellowship from the K.Lisa Yang Integrative Computational Neuroscience center. He obtained his PhD in Electrical and Computer Engineering from Johns Hopkins University. His research interests are at the intersection of the science of deep learning, signal processing, and optimization.