Bridging Sub-Tasks to Long-Horizon Task in Hierarchical Goal-Based Reinforcement Learning

Abstract

Hierarchical goal-based reinforcement learning (HGRL) is a promising approach to learning a long-horizon task by decomposing it into a series of subtasks of achieving subgoals in a shorter horizon. However, the performance of HGRL crucially depends on the design of intrinsic rewards for these subtasks: as frequently observed in prac- tice, short-sighted reward designs often lead the agent into undesirable states where the final goal is no longer achievable. One potential remedy to the issue is to provide the agent with a means to evaluate the achievability of the final goal upon the com- pletion of the subtask; yet, evaluating this achievability over a long planning horizon is a challenging task by itself. In this work, we propose a subtask reward scheme to bridge the gap between the long-horizon primary goal and short-horizon subtasks by incorporating look-ahead information towards the next subgoals. We provide an extensive empirical analysis in MuJoCo environments, demonstrating the importance of looking ahead to the subsequent sub-goals and improving the proposed framework applied to the existing HGRL baselines.