PGMT: Perceptive General Motion Tracking for Humanoid Robots

A unified perceptive policy for terrain-adaptive locomotion,
whole-body motion tracking, and teleoperation.

Hongyi Li¹Peizhuo Li²Yucheng Tao¹Ze Wang¹Fangzhou Xu²Jinyi Chen¹Yanyan Yuan³Dapeng Jia²Yongbin Jin^3,†Mingfeng Fan^2,†Guillaume Sartoretti²Hongtao Wang^1,3,‡

¹ Zhejiang University² National University of Singapore³ MirrorMe Technology

† Corresponding authors ‡ Project lead

Paper arXiv Video Bilibilisoon BibTeX

PGMT executes diverse whole-body motions across stairs, uneven terrain, and outdoor environments. A single policy supports locomotion, motion tracking, teleoperation, and fall recovery.

Abstract

TL;DR

PGMT learns terrain adaptation within a general motion tracker, enabling one policy to execute diverse motion commands across complex real-world terrain.

Humanoid motion trackers can reproduce diverse whole-body motions, but their performance degrades on complex terrain where terrain-agnostic references become physically infeasible. We present PGMT, a Perceptive General Motion Tracking pipeline for humanoid robots that learns terrain adaptation from independently selected motion references and terrains. PGMT first learns a general tracking and recovery prior, then incorporates terrain perception through motion-conditioned terrain glimpses that selectively encode regions relevant to the current motion. Terrain-aware tracking relaxation allows necessary deviations from the reference while preserving its motion intent.

Zero-shot deployment on a Unitree G1 demonstrates robust terrain-adaptive locomotion and whole-body motion execution over real-world terrain with obstacles up to 37 cm high, while supporting teleoperation, dynamic motion tracking, and fall recovery. PGMT extends general humanoid motion tracking beyond flat ground, providing a unified policy for terrain-adaptive locomotion, diverse whole-body behaviors, and teleoperation in complex environments.

Method Overview

A general motion prior is progressively extended with terrain perception,
enabling adaptation without terrain-matched reference trajectories.

Stage 1

Tracking pretraining

Learn diverse whole-body tracking and recovery on flat ground. An Intent Fusion Module combines robot history with future reference motions, while a split-return critic separates the training objectives.

Stage 2

Perception injection

Introduce four motion-conditioned terrain glimpses and terrain-contact objectives. Terrain-aware tracking relaxation permits lower-body adaptation while preserving upper-body motion intent.

The resulting policy supports multiple command sources through a shared whole-body control interface. Read the method in the paper

Real-World Demonstrations

A single PGMT policy deployed zero-shot on a 29-DoF Unitree G1.
All demonstrations use the same policy, without switching controllers.

Clips are shown at their original playback speed. Select a category to explore additional demonstrations.

Onboard systemUnitree G1Livox Mid-360S LiDARNVIDIA Jetson Orin NX

BibTeX

@misc{li2026pgmt,
  title   = {PGMT: Perceptive General Motion Tracking for Humanoid Robots},
  author  = {Li, Hongyi and Li, Peizhuo and Tao, Yucheng and Wang, Ze and
             Xu, Fangzhou and Chen, Jinyi and Yuan, Yanyan and Jia, Dapeng and
             Jin, Yongbin and Fan, Mingfeng and Sartoretti, Guillaume and Wang, Hongtao},
  year    = {2026},
  eprint  = {2609.08511},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url     = {https://arxiv.org/abs/2609.08511}
}

arXiv preprint · arXiv:2609.08511 · 2026.