👏🏻 CLAP : Contrastive Latent Action Pretraining

Learning Vision-Language-Action Models from Human Videos

Chubin Zhang^1,2,* Jianan Wang^2,* Zifeng Gao¹ Yue Su^2,3 Tianru Dai¹ Cai Zhou⁴ Jiwen Lu¹ Yansong Tang¹

¹Tsinghua University ²Astribot ³University of Hong Kong ⁴MIT

^*Equal Contribution ✉ Corresponding Author

TL;DR

CLAP aligns visual latent actions from human videos with proprioceptive latent actions from robot trajectories through contrastive learning, enabling effective skill transfer from abundant human demonstrations to robotic execution.

Abstract

Generalist Vision-Language-Action models are currently hindered by the scarcity of robotic data compared to the abundance of human video demonstrations. Existing Latent Action Models attempt to leverage video data but often suffer from visual entanglement, capturing noise rather than manipulation skills.

To address this, we propose Contrastive Latent Action Pretraining (CLAP), a framework that aligns the visual latent space from videos with a proprioceptive latent space from robot trajectories. By employing contrastive learning, CLAP maps video transitions onto a quantized, physically executable codebook. Building on this representation, we introduce a dual-formulation VLA framework offering both CLAP-NTP, an autoregressive model excelling at instruction following and object generalization, and CLAP-RF, a Rectified Flow-based policy designed for high-frequency, precise manipulation.

Furthermore, we propose a Knowledge Matching (KM) regularization strategy to mitigate catastrophic forgetting during fine-tuning. Extensive experiments demonstrate that CLAP significantly outperforms strong baselines, enabling the effective transfer of skills from human videos to robotic execution.

Method

Contrastive Latent Action Learning

Unlike conventional methods that rely solely on limited robot teleoperation data, CLAP learns an executable latent action space from large-scale human demonstrations. We align visual dynamics from videos with proprioceptive dynamics from robot trajectories through contrastive learning.

Dual-Formulation VLA

CLAP-NTP retains autoregressive architecture for strong reasoning and instruction-following. CLAP-RF uses Rectified Flow for high-frequency inference (183ms on RTX 3090) with exceptional precision.

Knowledge Matching Regularization

Our KM strategy mitigates catastrophic forgetting during fine-tuning, preserving the semantic knowledge from pretraining while adapting to new tasks.

Real World Experiments

CLAP-NTP Pick and Place — Demonstrates strong instruction following capabilities with precise object manipulation. The autoregressive model preserves semantic reasoning from the VLM backbone. 2× speedup

CLAP-NTP Make Bouquet — Successfully transfers manipulation skills to novel objects by learning from human video demonstrations. This showcases CLAP's core capability: enabling object generalization through contrastive alignment. 2× speedup

CLAP-RF Cloth Folding — Demonstrates precise manipulation in delicate tasks requiring fine motor skills. The Rectified Flow-based policy enables high-frequency control essential for contact-rich manipulation. 2× speedup

CLAP-RF Gift Packing — Complex multi-step manipulation showcasing dexterous bimanual control. CLAP-RF outperforms strong baselines like π₀ and π₀.₅ in tasks requiring exceptional precision. Generalizes directly to novel objects. 2× speedup

Key Results

+25%

Success Rate Improvement

fine-tuning with human videos

183ms

Inference Latency

CLAP-RF on RTX 3090

Robot Platforms

Astribot, AgiBot, Franka

Citation

@article{zhang2026clap,
  title={CLAP: Contrastive Latent Action Pretraining for 
         Learning Vision-Language-Action Models from Human Videos},
  author={Zhang, Chubin and Wang, Jianan and Gao, Zifeng and Su, Yue and Dai, Tianru 
          and Zhou, Cai and Lu, Jiwen and Tang, Yansong},
  journal={arXiv preprint arXiv:2601.04061},
  year={2026}
}