Selective dropout for deep neural networks

Erik Barrow; Mark Eastwood; Chrisina Jayne

doi:10.1007/978-3-319-46675-0_57

Selective dropout for deep neural networks

Erik Barrow, Mark Eastwood, Chrisina Jayne

Centre for Digital Innovation

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

2 Citations (Scopus)

Abstract

Dropout has been proven to be an effective method for reducing overfitting in deep artificial neural networks. We present 3 new alternative methods for performing dropout on a deep neural network which improves the effectiveness of the dropout method over the same training period. These methods select neurons to be dropped through statistical values calculated using a neurons change in weight, the average size of a neuron’s weights, and the output variance of a neuron. We found that increasing the probability of dropping neurons with smaller values of these statistics and decreasing the probability of those with larger statistics gave an improved result in training over 10, 000 epochs. The most effective of these was found to be the Output Variance method, giving an average improvement of 1.17% accuracy over traditional dropout methods.

Original language	English
Title of host publication	Neural Information Processing - 23rd International Conference, ICONIP 2016, Proceedings
Editors	Akira Hirose, Minho Lee, Derong Liu, Kenji Doya, Kazushi Ikeda, Seiichi Ozawa
Publisher	Springer-Verlag
Pages	519-528
Number of pages	10
ISBN (Print)	9783319466743
DOIs	https://doi.org/10.1007/978-3-319-46675-0_57
Publication status	Published - 2016
Event	13th International Conference on Neural Information Processing Systems - Barcelona, Spain Duration: 5 Dec 2016 → 10 Dec 2016

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	9949 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	13th International Conference on Neural Information Processing Systems
Country/Territory	Spain
City	Barcelona
Period	5/12/16 → 10/12/16

Bibliographical note

Publisher Copyright:
© Springer International Publishing AG 2016.

Copyright:
Copyright 2017 Elsevier B.V., All rights reserved.

Access to Document

10.1007/978-3-319-46675-0_57

Cite this

Barrow, E., Eastwood, M., & Jayne, C. (2016). Selective dropout for deep neural networks. In A. Hirose, M. Lee, D. Liu, K. Doya, K. Ikeda, & S. Ozawa (Eds.), Neural Information Processing - 23rd International Conference, ICONIP 2016, Proceedings (pp. 519-528). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9949 LNCS). Springer-Verlag. https://doi.org/10.1007/978-3-319-46675-0_57

Barrow, Erik ; Eastwood, Mark ; Jayne, Chrisina. / Selective dropout for deep neural networks. Neural Information Processing - 23rd International Conference, ICONIP 2016, Proceedings. editor / Akira Hirose ; Minho Lee ; Derong Liu ; Kenji Doya ; Kazushi Ikeda ; Seiichi Ozawa. Springer-Verlag, 2016. pp. 519-528 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{eade97070a234f499009816ed5bd82ef,

title = "Selective dropout for deep neural networks",

abstract = "Dropout has been proven to be an effective method for reducing overfitting in deep artificial neural networks. We present 3 new alternative methods for performing dropout on a deep neural network which improves the effectiveness of the dropout method over the same training period. These methods select neurons to be dropped through statistical values calculated using a neurons change in weight, the average size of a neuron{\textquoteright}s weights, and the output variance of a neuron. We found that increasing the probability of dropping neurons with smaller values of these statistics and decreasing the probability of those with larger statistics gave an improved result in training over 10, 000 epochs. The most effective of these was found to be the Output Variance method, giving an average improvement of 1.17% accuracy over traditional dropout methods.",

author = "Erik Barrow and Mark Eastwood and Chrisina Jayne",

note = "Publisher Copyright: {\textcopyright} Springer International Publishing AG 2016. Copyright: Copyright 2017 Elsevier B.V., All rights reserved.; 13th International Conference on Neural Information Processing Systems ; Conference date: 05-12-2016 Through 10-12-2016",

year = "2016",

doi = "10.1007/978-3-319-46675-0_57",

language = "English",

isbn = "9783319466743",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer-Verlag",

pages = "519--528",

editor = "Akira Hirose and Minho Lee and Derong Liu and Kenji Doya and Kazushi Ikeda and Seiichi Ozawa",

booktitle = "Neural Information Processing - 23rd International Conference, ICONIP 2016, Proceedings",

}

Barrow, E, Eastwood, M & Jayne, C 2016, Selective dropout for deep neural networks. in A Hirose, M Lee, D Liu, K Doya, K Ikeda & S Ozawa (eds), Neural Information Processing - 23rd International Conference, ICONIP 2016, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 9949 LNCS, Springer-Verlag, pp. 519-528, 13th International Conference on Neural Information Processing Systems, Barcelona, Spain, 5/12/16. https://doi.org/10.1007/978-3-319-46675-0_57

Selective dropout for deep neural networks. / Barrow, Erik; Eastwood, Mark; Jayne, Chrisina.
Neural Information Processing - 23rd International Conference, ICONIP 2016, Proceedings. ed. / Akira Hirose; Minho Lee; Derong Liu; Kenji Doya; Kazushi Ikeda; Seiichi Ozawa. Springer-Verlag, 2016. p. 519-528 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9949 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Selective dropout for deep neural networks

AU - Barrow, Erik

AU - Eastwood, Mark

AU - Jayne, Chrisina

PY - 2016

Y1 - 2016

N2 - Dropout has been proven to be an effective method for reducing overfitting in deep artificial neural networks. We present 3 new alternative methods for performing dropout on a deep neural network which improves the effectiveness of the dropout method over the same training period. These methods select neurons to be dropped through statistical values calculated using a neurons change in weight, the average size of a neuron’s weights, and the output variance of a neuron. We found that increasing the probability of dropping neurons with smaller values of these statistics and decreasing the probability of those with larger statistics gave an improved result in training over 10, 000 epochs. The most effective of these was found to be the Output Variance method, giving an average improvement of 1.17% accuracy over traditional dropout methods.

AB - Dropout has been proven to be an effective method for reducing overfitting in deep artificial neural networks. We present 3 new alternative methods for performing dropout on a deep neural network which improves the effectiveness of the dropout method over the same training period. These methods select neurons to be dropped through statistical values calculated using a neurons change in weight, the average size of a neuron’s weights, and the output variance of a neuron. We found that increasing the probability of dropping neurons with smaller values of these statistics and decreasing the probability of those with larger statistics gave an improved result in training over 10, 000 epochs. The most effective of these was found to be the Output Variance method, giving an average improvement of 1.17% accuracy over traditional dropout methods.

UR - http://www.scopus.com/inward/record.url?scp=84992699641&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-46675-0_57

DO - 10.1007/978-3-319-46675-0_57

M3 - Conference contribution

AN - SCOPUS:84992699641

SN - 9783319466743

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 519

EP - 528

BT - Neural Information Processing - 23rd International Conference, ICONIP 2016, Proceedings

A2 - Hirose, Akira

A2 - Lee, Minho

A2 - Liu, Derong

A2 - Doya, Kenji

A2 - Ikeda, Kazushi

A2 - Ozawa, Seiichi

PB - Springer-Verlag

T2 - 13th International Conference on Neural Information Processing Systems

Y2 - 5 December 2016 through 10 December 2016

ER -

Barrow E, Eastwood M, Jayne C. Selective dropout for deep neural networks. In Hirose A, Lee M, Liu D, Doya K, Ikeda K, Ozawa S, editors, Neural Information Processing - 23rd International Conference, ICONIP 2016, Proceedings. Springer-Verlag. 2016. p. 519-528. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-319-46675-0_57

Selective dropout for deep neural networks

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this