Fitting functions and their derivatives with neural networks

Abstract

Scope and Method of Study: The objective of this work was to study methods of simultaneously approximating functions and their first-order derivatives using multilayer feedforward neural networks. There are a few methods proposed today to simultaneously approximate both functions and their first derivatives, but they require modifications to the network structure. The new method works with any multilayer feedfoward neural network, by forming a new performance index that combines both the function error and the first derivative error. We tested and analyzed the results of the proposed method on both analytic and real-world problems.

Findings and Conclusions: We selected two optimization procedures for the new performance index. The first procedure was for any gradient-based optimization, while the other was implemented under the Levenberg-Marquardt framework. For each procedure, extra backpropagation calculations were derived to force the first derivative response of the neural network to match the desired derivative target. Moreover, we discovered two new types of overfitting from neural networks trained with the proposed performance index. We analyzed and illustrated how the overfitting develops. A network pruning algorithm was proposed to eliminate these types of overfitting. The simulation results tested on four analytic problems and three systems in Molecular Dynamics consistently showed that the approximation accuracy of neural networks trained by the new performance index significantly outperformed the use of standard training methods. In addition, the network generalization was even further improved with the incorporation of the pruning algorithm. We found that the most promising method yielding the most accurate approximation and the best generalization was to optimize the new performance index under the Levenberg-Marquardt framework along with the use of the pruning algorithm.