Primal-dual techniques for nonlinear programming and applications to artificial neural network training.

Abstract

In this work, new developments in primal-dual techniques for general constrained non-linear programming problems are proposed. We first implement a modified version of the general nonlinear primal-dual algorithm that was published by El-Bakry et al. (21). We use the algorithm as a backbone of a new stochastic hybrid technique for solving general constrained nonlinear programming problems. The idea is to combine a fast local optimization strategy and a global search technique. The technique is a modified nonlinear primal-dual technique that uses concepts from simulated annealing to increase the probability of converging to the global minima of the objective function. At each iteration, the algorithm solves the Karush-Kuhn-Tucker optimality conditions to find the next iterate. A random noise is added to the resulting direction of move in order to escape local minima. The noise is gradually removed throughout the iteration process. We show that for complicated problems that possess numerous local minima and global minima, the proposed algorithm outperforms the deterministic approach. We also develop a new class of incremental nonlinear primal-dual techniques for solving optimization problems with special decomposition properties. Specifically, the objective functions of the problems are sums of independent nonconvex differentiable terms minimized subject to a set of nonlinear constraints for each term. The technique performs successive primal-dual increments for each decomposition term of the objective function. The method is particularly beneficial for online applications and problems that have a large amount of data. We show that the technique can be nicely applied to artificial neural training and provide experimental results for financial forecasting problems.