Transforming the search space for finding diverse policices

What is it about?

Policy Manifold Search is an approach to performing diversity-based policy search, which leverages the concept of the manifold hypothesis, in the context of the policy network parameter space. Instead of performing policy search in the original high-dimensional space of policy's neural network parameters, we find a lower-dimensional representation which preserves the properties of the original parameter space, and perform the search in this representation space. MAP-Elites framework is used to maintain a collection of policies which exhibit diverse behaviours in the environment, and serves both as a dataset for learning policy parameter representations, as well as a principled framework to do diversity-based policy search. The findings from our experimental evaluations suggest that using a lower-dimensional policy parameter representation helps improve sample efficiency and the overall number of discovered behaviours compared to the state-of-the-art approaches.

Featured Image

Photo by Possessed Photography on Unsplash

Photo by Possessed Photography on Unsplash

Why is it important?

The proposed approach brings two main benefits: (i) by producing the parameter space representations, the original high-dimensional search space is reduced, which leads to improved sample-efficiency of the diversity-based policy search. (ii) based on the insights about the dataset generation process and manifold learning models, a framework for further study of the parameter space properties is introduced.

Perspectives