A Framework for Quantifying Autonomy in Robotic Systems

ABSTRACT: Although autonomous functioning facilitates the deployment of robotic systems in operating domains that support limited to no human oversight, establishing correspondence between task requirements and a system’s autonomous performance is still an open challenge. Several techniques for characterizing operating domains and/or quantifying autonomy have been proposed over the last three decades, however, to our knowledge, these have no discernment of sub-mode features of variation of autonomy, and some are based on metrics that are susceptible to the Goodhart’s law. This paper introduces a capability-based quantitative autonomy assessment framework for fully autonomous systems. The formulation of the framework started by establishing robot task characteristics from which three autonomy metrics, namely an essential capability set, reliability, and responsiveness, were derived. The characteristics were founded on the realization that robots ultimately replace human skilled workers, from which a relationship between human job and robot task characteristics was established. Additionally, mathematical formulations relating metrics to autonomy are also presented. To emphasize the fact that autonomy is not just a question of existence, but also one of performance of a capability, the framework represents it as a two-part measure, of level and degree of autonomy. Usage of the framework has been demonstrated on two case studies, namely an autonomous vehicle at an on-road dynamic driving task and the DARPA Subterranean Challenge analysis. The framework provides not only a tool for quantifying autonomy and monitoring the integrity of systems, but also a regulatory interface and common language for autonomous systems’ developers and users.