We empirically decompose the generalization loss of deep neural networks into bias and variance components on an image classification task by constructing ensembles using geometric-mean averaging of the sub-net outputs and we isolate double descent at the variance component of the loss. Our results show that small models afford ensembles that outperform single large models while requiring considerably fewer parameters and computational steps. We also find that deep double descent that depends on the existence of label noise can be mitigated by using ensembles of models subject to identical label noise almost as thoroughly as by ensembles of networks each trained subject to i.i.d. noise.
The Institut für Neuroinformatik (INI) is a central research unit of the Ruhr-Universität Bochum. We aim to understand the fundamental principles through which organisms generate behavior and cognition while linked to their environments through sensory systems and while acting in those environments through effector systems. Inspired by our insights into such natural cognitive systems, we seek new solutions to problems of information processing in artificial cognitive systems. We draw from a variety of disciplines that include experimental approaches from psychology and neurophysiology as well as theoretical approaches from physics, mathematics, electrical engineering and applied computer science, in particular machine learning, artificial intelligence, and computer vision.