Selective dropout for deep neural networks

Erik Barrow, Mark Eastwood, Chrisina Jayne

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)

Abstract

Dropout has been proven to be an effective method for reducing overfitting in deep artificial neural networks. We present 3 new alternative methods for performing dropout on a deep neural network which improves the effectiveness of the dropout method over the same training period. These methods select neurons to be dropped through statistical values calculated using a neurons change in weight, the average size of a neuron’s weights, and the output variance of a neuron. We found that increasing the probability of dropping neurons with smaller values of these statistics and decreasing the probability of those with larger statistics gave an improved result in training over 10, 000 epochs. The most effective of these was found to be the Output Variance method, giving an average improvement of 1.17% accuracy over traditional dropout methods.

Original languageEnglish
Title of host publicationNeural Information Processing - 23rd International Conference, ICONIP 2016, Proceedings
EditorsAkira Hirose, Minho Lee, Derong Liu, Kenji Doya, Kazushi Ikeda, Seiichi Ozawa
PublisherSpringer-Verlag
Pages519-528
Number of pages10
ISBN (Print)9783319466743
DOIs
Publication statusPublished - 2016
Event13th International Conference on Neural Information Processing Systems - Barcelona, Spain
Duration: 5 Dec 201610 Dec 2016

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume9949 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference13th International Conference on Neural Information Processing Systems
CountrySpain
CityBarcelona
Period5/12/1610/12/16

Bibliographical note

Publisher Copyright:
© Springer International Publishing AG 2016.

Copyright:
Copyright 2017 Elsevier B.V., All rights reserved.

Fingerprint Dive into the research topics of 'Selective dropout for deep neural networks'. Together they form a unique fingerprint.

Cite this