OPTIMIZATION BASED MARKOV MODEL REDUCTIONS

Abstract

Markov Decision Processes (MDPs) are well-studied and flexible structures that modela wide-range of phenomena, but are often subject to an exponentially increasing state space size, thus rendering exact methods intractable on large-scale instances. Existing literature leverages a variety of assumptions to create simplified structures or aggregations to transfer intractable MDPs into the realm of computability. This research bridges the gap of the scalability problem with interpretability through consideration of bilevel and integer programming approaches supported by orthodox optimization theory and practice. Five models are constructed under different state space reduction perspectives of which two are proved to hold equivalence at optimality. This equivalence links the domain of the more natural, but also more computationally challenging, proposed bilevel programs to a mixed-integer linear program (MILP) supported by established theoretical results within the existing model reduction space. The bilevel models are reformulated into single level equivalents and are compared alongside the MILP program through a small scale illustrative example along with computer experimentation to gauge runtime and scalability potential. Holding the most promise from the experimentation phase, the MILP formulation is extended through derivation and testing of both a novel warm start heuristic as well as a set of symmetry breaking constraints for cardinality constraint subclasses of MDP problem instances. Finally, the applications of the research is discussed with connections to problem classes with established state costs and industry domains with requirements for policy consolidation and interpretation.