Development and validation of metrics to evaluate robotics operator performance.

Abstract

This dissertation sought to develop and validate objective performance metrics for RMS operators to facilitate operator screening, provide training feedback, and examine performance changes over time. Preliminary investigation identified smooth hand controller inputs (ramping) and multi-axis commanding as skills that are critical to the safe, effective operation of the RMS robotic arm. Theoretical models of expert performance were developed that helped to identify critical aspects of these skills. Performance metrics were selected that quantified the differences observed during comparisons of actual operator performance with the theoretical models. Observed control strategies were discussed from a human factors standpoint.

Validation of the metrics was achieved in an empirical study during which twelve novice operators performed a series of targeted movement tasks designed to evaluate ramping performance and the use of multi-axis control. Ramping performance was assessed by examining velocities and accelerations, R-square values (representing smoothness of the commanded inputs), completion times, and distances traveled during the ramp-in, travel, ramp-out, and correction phases of the task. Multi-axis control was assessed by examining percentages of control usage (single, dual, and triple-axis), lag times, completion times, accelerations, and correction times. MANOVA results indicated that ramping performance was significantly affected by movement distance (F 7,5 = 333.02, p < .0001) and operator ( F77,103.32 = 6.86, p < .0001), and multi-axis performance varied significantly due to task distance ratios ( F4,8 = 22.63, p = .0002), replicate ( F4,8 = 35.87, p < .0001), and operator (F44,1727.4 = 19.85, p < .0001). A select subset of the metrics successfully achieved a reasonable classification of RMS operator performance.

Remote Manipulator Systems (RMS) aboard the International Space Station and the Space Shuttle assist in payload management, crew movement, retrieval of objects from space, and station construction. Generic Robotics Training (GRT) uses the Basic Operational Robotics Instructional System (BORIS) graphics simulator to teach basic robotics concepts and skills that are transferable to any of the on-orbit RMS systems. Currently, operator competency is assessed on a five-point scale by an "expert" observer. There is a need for quantitative performance metrics that would achieve an objective, reliable, and sensitive evaluation of RMS operator capabilities.