Latent Potential in Mixture-of-Experts for Improving Interpretability, Training Processes, and Dynamic Inference

Abstract

In the field of artificial intelligence and machine learning (AI/ML), neural networks are a common and powerful statistical modeling tool. The neural architecture of such models describes how information is processed and decisions are made. The \textit{Mixture-of-Experts} neural architecture is a metacognitive methodology applying the divide-and-conquer strategy to AI/ML. A \textit{router} model learns ``learns how to learn'' where it defines subproblem decision boundaries (divide), whereby cognitive submodels (experts) are specialized (conquer). Thus, experts collectively provide a solution to the full problem space and can be passed samples independently or collaborate together. The research questions explored in this dissertation explore leveraging the router and its understanding of domain meta-characteristics to address issues such as neural interpretability, training efficiency, and computation optimization. This work applies the MNIST, CIFAR-10, and CIFAR-100 image classification problem spaces to evaluate approaches to these issues and simultaneously provide metrics for common benchmarking datasets. Interpretability is explored through the lenses of various 2D projection algorithms and attention-based routing to improve human understanding of model decisions, such as how subproblem regions are formed, which input features are relevant, and the behavioral relationships between experts. Training efficiency is addressed through transfer learning, where pretrained autoencoder and classifier models are adapted as router models, though experimental results fail to prove these strategies effective. Computation optimization is approached here as a complexity imbalance, where some subproblems are more difficult than others. Scaling up experts to handle increasingly complex subproblems enables dynamic inference, improving overall system performance without further increasing computational costs when handling simpler inputs.