• RUB
  • INI
  • Projects
  • Combat Task Interference in Multi-Task Model-Based Reinforcement Learning by Using Separate Dynamics Models
2019
2021
Combat Task Interference in Multi-Task Model-Based Reinforcement Learning by Using Separate Dynamics Models

In model-free multi-task reinforcement learning (RL), abundant work shows that a shared policy network can improve performance across the different tasks. The rationale behind this is that an agent can learn similarities that all tasks have in common and thus effectively enrich the sample count for all tasks at hand. In model-based multi-task RL however, we found evidence suggesting that a dynamics model can suffer from task confusion or catastrophic interference if it is trained on multiple tasks at once.

To mitigate this problem, we use a set of distinct dynamics models which are orchestrated by a task classification network. The latter is trained to detect the task at hand and effectively picks one of the dynamics models for planning. Once a dynamics model is chosen, we use it to generate imaginary rolluts to determine the agent's next actions in the real environment. By separating the different task dynamics in different networks, we avoid catastrophic interference to a large degree, which leads to higher quality and more stable rollouts for our proposed approach. When using only a single dynamics model for all tasks, we found clear evidence of task confusion. The dynamics model was prone to randomly switching tasks in the middle of a rollout, which rendered the rollout unusable for planning. We conjecture that in contrast to learning a shared policy, for learning the task dynamics it is more important to model the details of the individual tasks correctly, since this is where they differ from each other. 

The Institut für Neuroinformatik (INI) is a central research unit of the Ruhr-Universität Bochum. We aim to understand the fundamental principles through which organisms generate behavior and cognition while linked to their environments through sensory systems and while acting in those environments through effector systems. Inspired by our insights into such natural cognitive systems, we seek new solutions to problems of information processing in artificial cognitive systems. We draw from a variety of disciplines that include experimental approaches from psychology and neurophysiology as well as theoretical approaches from physics, mathematics, electrical engineering and applied computer science, in particular machine learning, artificial intelligence, and computer vision.

Universitätsstr. 150, Building NB, Room 3/32
D-44801 Bochum, Germany

Tel: (+49) 234 32-28967
Fax: (+49) 234 32-14210