Interpreting Categorical Distributional Reinforcement Learning: An Implicit Risk-Sensitive Regularization Effect

TL;DR — We interpret distributional reinforcement learning from the perspective of regularization effect.

Abstract

The theoretical advantages of distributional reinforcement learning~(RL) over expectation-based RL remain elusive, despite its remarkable empirical performance. Starting from Categorical Distributional RL~(CDRL), our work attributes the potential superiority of distributional RL to its \textit{risk-sensitive entropy regularization}. This regularization stems from the additional return distribution information regardless of only its expectation via the return density function decomposition, a variant of the gross error model in robust statistics. Compared with maximum RL that explicitly optimizes the policy to encourage the exploration, we reveal that the resulting risk-sensitive entropy regularization of CDRL plays a different role as an augmented reward function. It implicitly optimizes policies for a risk-sensitive exploration towards true target return distributions, which helps to reduce the intrinsic uncertainty of the environment. Finally, extensive experiments verify the importance of this risk-sensitive regularization in distributional RL, as well as the mutual impacts of both explicit and implicit entropy regularization.