Automatic Machine Learning in Optimization and Data Augmentation

Abstract

This dissertation introduces Automatic Machine Learning (AutoML) as a potential approach to overcome current deep learning challenges on efficiency and cost. It also proposes two novel AutoML workflows to areas in deep learning where AutoML is less recognized by the AI community: optimization and data augmentation.

The proposed AutoML workflow in optimization can automatically adjust the learning rate for deep learning tasks. It monitors the signals generated during optimization and dynamically changes the learning rate based on the signal observed. The workflow is successfully deployed in image classification, instance detection and language modeling tasks. The method delivers better performance and faster convergence speed than widely-used static and learning-based schedulers under various different settings.

The new AutoML workflow in data augmentation can help deep neural networks achieve better generalization performance through automated optimization of data augmentation policies. Comparing with the prior best method, the workflow halves the computation required while achieving equivalent or better results on the same benchmarks. In addition, it also removes the need of human intervention in the workflow, making the workflow truly automated for deep learning applications.

Finally, the dissertation concludes that AutoML can play a significant role on various aspects of deep learning through further efficiency improvement and cost reduction. The hope of the dissertation is to inspire more AutoML research on all areas of deep learning, so that AutoML can eventually facilitate the development of fully automated learning, an important milestone in our long pursuit of Artificial Intelligence that can lead us to a brighter future.