Relational Deep Reinforcement Learning for Autonomous Sensor Control

Abstract

Rapid improvements in processing hardware and sensing technologies have enabled the development of autonomous systems across countless domains. To fully realize the potential of these systems, sensor control algorithms must efficiently leverage limited sensing resources to meet task objectives while satisfying latency constraints for real-time operation. Traditional techniques based on optimal control theory and classical signal processing often struggle to meet these demands due to their high computational requirements and poor scalability. Existing learning-based techniques are computationally lightweight but frequently fail to generalize in unseen scenarios. Instead, this dissertation proposes that deep reinforcement learning (RL), relational reasoning, and continuous control are powerful tools that can be combined to solve a large class of sensor control problems. This hypothesis is explored in two increasingly relevant application spaces: spectrum allocation between radar and communications devices and sensor management for multi-object search and track tasks. In both cases, the relational reasoning techniques outperform state-of-the-art methods from the literature in terms of task performance, runtime complexity, or both. The first case study focuses on the radio frequency (RF) spectrum sharing problem from a radar perspective. The growing demand for RF spectrum has placed considerable strain on radar systems. Future radar systems must be designed with coexistence in mind to avoid harmful mutual interference that compromises the quality of service for other users in the channel. This work presents a deep RL approach to spectrum sharing that enables a pulse-agile cognitive radar to operate in congested spectral environments. The deep RL approach is shown to outperform several techniques from the literature while generalizing across diverse scenarios in a software-defined radio (SDR) testbed. The second half of this dissertation focuses on sensor control for the multi-object search and track problem. In these tasks, an autonomous platform must search a region of interest for unknown objects while maintaining accurate state estimates on detected objects. Existing solutions to this problem rely on computationally intensive online planning routines that scale poorly to complex, high-dimensional control spaces. This work develops a graph-theoretic solution that leverages recent advances in relational reasoning and model-based RL. Experimental results demonstrate that the graph RL approach outperforms state-of-the-art planning techniques in terms of tracking performance while reducing execution times by several orders of magnitude. This makes the proposed approach highly suitable for real-time operation on resource-constrained platforms.