publication . Preprint . 2019

Positional Normalization

Li, Boyi; Wu, Felix; Weinberger, Kilian Q.; Belongie, Serge;
Open Access English
  • Published: 09 Jul 2019
Abstract
A popular method to reduce the training time of deep neural networks is to normalize activations at each layer. Although various normalization schemes have been proposed, they all follow a common theme: normalize across spatial dimensions and discard the extracted statistics. In this paper, we propose an alternative normalization method that noticeably departs from this convention and normalizes exclusively across channels. We argue that the channel dimension is naturally appealing as it allows us to extract the first and second moments of features extracted at a particular image position. These moments capture structural information about the input image and ex...
Subjects
free text keywords: Computer Science - Computer Vision and Pattern Recognition, Computer Science - Machine Learning
Funded by
NSF| CAREER: New Directions for Metric Learning
Project
  • Funder: National Science Foundation (NSF)
  • Project Code: 1149882
  • Funding stream: Directorate for Computer & Information Science & Engineering | Division of Information and Intelligent Systems
,
NSF| TRIPODS: Data Science for Improved Decision-Making: Learning in the Context of Uncertainty, Causality, Privacy, and Network Structures
Project
  • Funder: National Science Foundation (NSF)
  • Project Code: 1740822
  • Funding stream: Directorate for Computer & Information Science & Engineering | Division of Computing and Communication Foundations
,
NSF| S&AS: INT: Inference, Reasoning, and Learning for Robust Autonomous Driving
Project
  • Funder: National Science Foundation (NSF)
  • Project Code: 1724282
  • Funding stream: Directorate for Computer & Information Science & Engineering | Division of Information and Intelligent Systems
,
NSF| III: Small: Collaborative Research: Towards Interpretable Machine Learning
Project
  • Funder: National Science Foundation (NSF)
  • Project Code: 1526012
Download from

DenseBlock 2 ResNet-18 Train ResNet-18 Val ResNet-18 + PONO Train ResNet-18 + PONO Val

Abstract
A popular method to reduce the training time of deep neural networks is to normalize activations at each layer. Although various normalization schemes have been proposed, they all follow a common theme: normalize across spatial dimensions and discard the extracted statistics. In this paper, we propose an alternative normalization method that noticeably departs from this convention and normalizes exclusively across channels. We argue that the channel dimension is naturally appealing as it allows us to extract the first and second moments of features extracted at a particular image position. These moments capture structural information about the input image and ex...
Subjects
free text keywords: Computer Science - Computer Vision and Pattern Recognition, Computer Science - Machine Learning
Funded by
NSF| CAREER: New Directions for Metric Learning
Project
  • Funder: National Science Foundation (NSF)
  • Project Code: 1149882
  • Funding stream: Directorate for Computer & Information Science & Engineering | Division of Information and Intelligent Systems
,
NSF| TRIPODS: Data Science for Improved Decision-Making: Learning in the Context of Uncertainty, Causality, Privacy, and Network Structures
Project
  • Funder: National Science Foundation (NSF)
  • Project Code: 1740822
  • Funding stream: Directorate for Computer & Information Science & Engineering | Division of Computing and Communication Foundations
,
NSF| S&AS: INT: Inference, Reasoning, and Learning for Robust Autonomous Driving
Project
  • Funder: National Science Foundation (NSF)
  • Project Code: 1724282
  • Funding stream: Directorate for Computer & Information Science & Engineering | Division of Information and Intelligent Systems
,
NSF| III: Small: Collaborative Research: Towards Interpretable Machine Learning
Project
  • Funder: National Science Foundation (NSF)
  • Project Code: 1526012
Download from

DenseBlock 2 ResNet-18 Train ResNet-18 Val ResNet-18 + PONO Train ResNet-18 + PONO Val

Powered by OpenAIRE Open Research Graph
Any information missing or wrong?Report an Issue