Automatic design of deep neural network architectures with evolutionary computation

Abstract

Deep Neural Networks (DNNs) are algorithms with widespread use in the extraction of knowledge from raw data. DNNs are used to solve problems in the fields of computer vision, natural language understanding, signal processing, and others. DNNs are state-of-the-art machine learning models capable of achieving better results than humans in many tasks. However, their application in fields outside computer science and engineering has been hindered due to the tedious process of trial and error multiple computationally intensive models. Thus, the development of algorithms that could allow for the automatic development of DNNs would further advance the field. Two central problems need to be addressed to allow the automatic design of DNN models: generation and pruning. The automatic generation of DNN architectures would allow for the creation of state-of-the-art models without relying on knowledge from human experts. In contrast, the automatic pruning of DNN architectures would reduce the computational complexity of such models for use in less powerful hardware. The generation and pruning of DNN models can be seen as a combinatorial optimization problem, which can be solved with the tools from the Evolutionary Computation (EC) field. This Ph.D. work proposes the use of Particle Swarm Optimization (PSO) for DNN architecture searching with competitive results and fast convergence, called psoCNN. Another algorithm based on Evolution Strategy (ES) is used for the pruning of DNN architectures, called DeepPruningES. The proposed psoCNN algorithm is capable of finding CNN architectures, a particular type of DNN, for image classification tasks with comparable results to human-crafted DNN models. Likewise, the DeepPruningES algorithm is capable of reducing the number of floating operations of a given DNN model up to 80 percent, and it uses the principles of Multi-Criteria Decision Making (MCDM) to output three pruned model with different trade-offs between computational complexity and classification accuracy. These ideas are then applied to the creation of a unified framework for searching highly accurate, and compact DNN applied for Medical Imaging Diagnostics, and the pruning of Generative Adversarial Networks (GANs) for Medical Imaging Synthesis with competitive results.