AN IMPROVED DESIGN FOR BACK PROPAGATION NEURAL NETWORKS USING A HIERARCHAL SUBNET DESIGN (HSD) APPROACH

Publication Summary

The common approach to the sizing and train- ing of Back Propagation neural networks is some- what adhoc. The optimal architecture and number of processing elements necessary to efficiently rep- resent the important features ofa particular data set depends on the nature of the data. The rate of net- work convergence and its optimality depend on the presentation sequence of the data, the data features, and the network architecture. The Hierarchal Sub- net Design (HSD) approach described here is an attempt to address these factors in a structured way, In the HSD approach the network is built and trained in an automatic, systematic, hierarchal manner such that the coarse features of the data are learned first followed by the finer details. Experimental results show that this approach can result in a faster rate of convergence and a lower minimum error than is attained with networks built and trained in a con- ventional manner.

Country	United States
Language	English (United States)
Related Person(s)	(AUTHOR)  Orhan Karaali (AUTHOR)  William M. Kushner
Copyright	Motorola Inc. October 1994

---

Continue to Text Preview →

Page 1 of 7

0 M

MOTOROLA Technical Developments Volumb 23 October 1994

AN IMPROVED DESIGN FOR BACK PROPAGATION NEFRAL NETWORKS USING A HIERARCHAL SUBNET DESIGN (HSD)~APPROACH

by Orhan Karaali and William M. Kushner :

being trained on-line and thus prevents them From being adaptable to changes in operating conditions. This lack of on-line adaptability limits their usefulness in desirable applications' such as channel equaliza- tion, noise compensation; speech recognition, speech coding, and speech synthesis.

  Another problem with conventional neural net- work training methodology is that optimal conver- gence of the network to Ithe minimum error condi- tion cannot be guaranteed except in trivial cases. BP neural networks converge by a process of gradi- ent descent. The conventional BP training method, presenting the entire training database to the ml1 network at each training cycle, is prone to suboptimal convergence at local minima, especially for com- plex networks and complex data sets. The suboptimal convergence, besides not igiving the lowest error rate, requires additional training cycles and can be computationally inefficient.

  Another deficiency in current neural network architecture design methodology is the absence of formal methods for sizing a network to the informa- tion content of the training data. Current conven- tional procedures are generally ad hoc. This is espe- cially true when the total information content ofthe data is unknown, or when only a subset ofthe data's information needs to be modeled. The methods described herein seek toiaddress these problems,

ABSTRACT

  The common approach to the sizing and train- ing of Back Propagation neural networks is some- what adhoc. The optimal architecture and number of processing elements necessary to efficiently rep- resent the important features ofa particular data set depends on the nature of the data. The rate of net- work convergence and its optimality depend on the presentation sequence of the data, the data features, and the network architecture. The Hierarchal Sub- net Design (HSD) approach described here is an attempt to address these factors in a structured way, In the HSD approach the network is built and trained in an automatic, systematic, hierarchal manner such that the coarse features of the data are learned first followed by the finer details. Experimental results show that this approach can result in a faster rate of convergence and a lower minimum error than is attained with networks built and trained in a con- ventional manner.

INTRODUCTION

  A major difficulty in training back propagation (BP) neural networks is the computational effort nec- essary to achieve a specified level of network con- vergence. The amount of processing necessary depends both on the inherent structure of the data and neural network architecture used to represent

it. Data with complex features requires a denser net- work to represent the information within it than does simpler data. The number of...

Download This Document →

Back to Overview