Enhanced recurrent network training

Abstract

In this dissertation, we introduce new, more efficient, methods for training recurrent neural networks (RNNs). These methods are based on a new understanding of the error surfaces of RNNs that has been developed in recent years. These error surfaces contain spurious valleys that disrupt the search for global minima. The spurious valleys are caused by instabilities in the networks, which become more pronounced with increased prediction horizons. The new methods described in this dissertation increase the prediction horizons in principled way that enables the search algorithms to avoid the spurious valleys.

The work also presents a novelty sampling method for collecting new data wisely. The clustering method determining when an RNN is extrapolating. The extrapolation occurs when RNN operates outside the region spanned by the training set, adequate performance cannot be guaranteed. The new method presented in this dissertation used the clustering method for extrapolation detection and collecting the novel data's. The training results are improved with the new data set by retraining the RNN.

The Model Reference control is introduced in this dissertation. The MRC is implemented on the simulated and experimental magnetic levitation system.