Controllable Human-Object
Interaction Synthesis

1Stanford University,2FAIR, Meta
arxiv 2023

CHOIS takes an initial object and human state, a language description and sparse object waypoints as input to synthesize both object motion and human motion. Using the waypoints extracted from 3D scenes as input conditions, CHOIS enables interaction synthesis in contextual environments.


Synthesizing semantic-aware, long-horizon, human-object interaction is critical to simulate realistic human behaviors. In this work, we address the challenging problem of generating synchronized object motion and human motion guided by language descriptions in 3D scenes.

We propose Controllable Human-Object Interaction Synthesis (CHOIS), an approach that generates object motion and human motion simultaneously using a conditional diffusion model given a language description, initial object and human states, and sparse object waypoints. While language descriptions inform style and intent, waypoints ground the motion in the scene and can be effectively extracted using high-level planning methods.

Naively applying a diffusion model fails to predict object motion aligned with the input waypoints and cannot ensure the realism of interactions that require precise hand-object contact and appropriate contact grounded by the floor. To overcome these problems, we introduce an object geometry loss as additional supervision to improve the matching between generated object motion and input object waypoints. In addition, we design guidance terms to enforce contact constraints during the sampling process of the trained diffusion model.


Method Overview

CHOIS Method Overview.



Results on Unseen Objects

Results on 3D Scenes