Cohere Labs - Esraa Elelimy, PhD student at the RLAI lab

Achieving fast and stable off-policy learning in deep reinforcement learning (RL) is challenging. Most existing methods rely on semi-gradient temporal-difference (TD) methods for their simplicity and efficiency, but are consequently susceptible to divergence. While more principled approaches like Gradient TD (GTD) methods have strong convergence guarantees, they have rarely been used in deep RL. Recent work introduced the Generalized Projected Bellman Error (GPBE), enabling GTD methods to work efficiently with nonlinear function approximation. However, this work is only limited to one-step methods, which are slow at credit assignment and require a large number of samples. In this paper, we extend the GPBE objective to support multistep credit assignment based on the λ-return and derive three gradient-based methods that optimize this new objective. We provide both a forward-view formulation compatible with experience replay and a backward-view formulation compatible with streaming algorithms. Finally, we evaluate the proposed algorithms and show that they outperform both PPO and StreamQ in MuJoCo and MinAtar environments, respectively.

Esraa is a third-year PhD student at the RLAI lab advised by Martha White. Her main research interests lie primarily in online reinforcement learning, real-time recurrent learning, and continual learning. She received her M.Sc. in Computing Science from the University of Alberta, where she worked on efficient real-time recurrent learning approaches for online reinforcement learning.

Add event to calendar

Apple Google Office 365 Outlook Outlook.com Yahoo