SkiLD: Unsupervised Skill Discovery
Guided by Factor Interactions

Zizhao Wang*, Jiaheng Hu*, Caleb Chuck*, Stephen Chen,
Roberto Martín-Martín, Amy Zhang, Scott Niekum, Peter Stone
The University of Texas at Austin, The University of Massachusetts Amherst, Sony AI
NeurIPS 2024
Paper Code

Overview

Unsupervised skill discovery carries the promise that an intelligent agent can learn reusable skills through autonomous, reward-free environment interaction. Existing unsupervised skill discovery methods learn skills by encouraging distinguishable behaviors that cover diverse states. However, in complex environments with many state factors (e.g., household environments with many objects), learning skills that cover all possible states is impossible, and naively encouraging state diversity often leads to simple skills that are not ideal for solving downstream tasks. This work introduces Skill Discovery from Local Dependencies (Skild), which leverages state factorization as a natural inductive bias to guide the skill learning process. The key intuition guiding Skild is that skills that induce diverse interactions between state factors are often more valuable for solving downstream tasks. To this end, Skild develops a novel skill learning objective that explicitly encourages the mastering of skills that effectively induce different interactions within an environment. We evaluate Skild in several domains with challenging, long-horizon sparse reward tasks including a realistic simulated household robot domain, where Skild successfully learns skills with clear semantic meaning and shows superior performance compared to existing unsupervised reinforcement learning methods that only maximize state coverage.

SkiLD describes skills that encode interactions (i.e., local dependencies) between state factors. In contrast to prior diversity-based methods that can easily get stuck by moving the robot to diverse, but non-interactive states, and factor-based methods that are trained to manipulate the hammer and nail, but not their interactions, SkiLD not only manipulate each object (left, middle) but also induce interactions between them (right), by specifying different local dependencies. These skills are often more useful than the ``easy'' skill learned by previous methods for downstream task-solving.

Skills Visualization

Just like other skill discovery methods, we can visualize SkiLD skills by randomly sampling a skill vector z and visualize the skill policy's behaviors conditioning on z. Here are some results:

Image 1
skill description
local dependencies
Image 2
grasp the peach
robot, peach → peach
Image 3
pick and place the knife
robot, sink, knife → knife
Image 3
cut the peach
robot, peach, knife → peach

iGibson

Image 1
skill description
local dependencies
Image 2
soak a rag in the sink
sink, rag → rag
Image 3
clean the car with a soaked rag
car, rag → rag

Mini-Behavior: Cleaning a Car

Image 1
skill description
local dependencies
Image 2
thaw fish in the sink
sink, fish → fish
Image 3
thaw olive in the sink
sink, olive → olive

Mini-Behavior: Thawing

Experiments

Skill Diversity

We evaluate whether SkiLD is indeed capable of achieving complex interaction graphs, comparing against two strong skill discovery baselines: DIAYN and CSD.
We unroll each of them for 500 episodes with randomly sampled skills z and examine the diversity of the interaction graphs they can induce.
The left figure illustrates the percentages of episodes where particular local dependencies have been induced at least once, in the Cleaning Car environment. We find that DIAYN and CSD are limited to skills that only manipulate one object individually, i.e. (agent, rag, action → rag) or (agent, soap, action → soap). By contrast, SkiLD learns to induce more complicated causal interactions, such as soaking the rag in the sink (sink, rag → rag) and cleaning the car with the soaked mug (car, rag → car).

Description of Image

These skills can then be used as low-level policy in a hierarchical RL setting, achieving significantly superior downstream task learning performance compared to previous state-of-the-art methods. Please checkout our paper and code for a detailed description of our methods and experiments.