What is it about?
This paper presents a survey of methods for pruning deep neural networks. It begins by categorising over 150 studies based on the underlying approach used and then focuses on three categories: methods that use magnitude based pruning, methods that utilise clustering to identify redundancy, and methods that use sensitivity analysis to assess the effect of pruning. Some of the key influencing studies within these categories are presented to highlight the underlying approaches and results achieved. Most studies present results which are distributed in the literature as new architectures, algorithms and data sets have developed with time, making comparison across different studied difficult. The paper therefore provides a resource for the community that can be used to quickly compare the results from many different methods on a variety of data sets, and a range of architectures, including AlexNet, ResNet, DenseNet and VGG. The resource is illustrated by comparing the results published for pruning AlexNet and ResNet50 on ImageNet and ResNet56 and VGG16 on the CIFAR10 data to reveal which pruning methods work well in terms of retaining accuracy whilst achieving good compression rates. The paper concludes by identifying some research gaps and promising directions for future research.
Featured Image
Why is it important?
Deep neural networks are excessively large and consume significant resources. Methods for reducing their size without compromising accuracy are therefore necessary, though there are many different approaches. This paper provides a comprehensive survey and insights emerging from the different algorithms.
Perspectives
One of the challenges in making sense of the empirical evaluations reported in the papers surveyed is that, as new deep learning architectures have developed and as new methods have been published, the comparisons carried out have evolved. The survey has therefore collated the published results of over 50 methods for a variety of data sets and architecture that is available as a resource for other researchers The comparison of published results shows that significant reductions can be obtained for AlexNet, ResNet and VGG, though there is no single method that is best, and that it is harder to prune ResNet than the other architectures. One can hypothesize that its use of skip connections makes it more optimal, though this is something that needs exploring. Likewise, given that different methods seem best for different architectures, it is worth studying and developing methods for specific architectures. The data also reveals that there are limited evaluations on other networks such as the InceptionNet, DenseNet, SegNet, FCN32 and datasets such as CIFAR100, Flowers102, CUB200-2011 A comprehensive independent evaluation of the methods that includes consideration of the issues raised by the Lottery hypothesis across a wider range of data and architectures would be a useful advance in the field.
Sunil Vadera
University of Salford
Read the Original
This page is a summary of: Methods for Pruning Deep Neural Networks, IEEE Access, January 2022, Institute of Electrical & Electronics Engineers (IEEE),
DOI: 10.1109/access.2022.3182659.
You can read the full text:
Contributors
The following have contributed to this page
