PAI-Actor: Cinematic Multi-Character Replacement in Dynamic Scenes

Bangxun Tang^1,2,*, Heyuan Gao^1,3,*, Yiren Song^1,4,*, Guian Fang^1,4, Zijian He¹, Jie Yang¹, Mike Zheng Shou^4,†
1Utopai Studios   ²University of California, Irvine   ³Nanyang Technological University   ⁴Show Lab, National University of Singapore
* Equal contribution   † Corresponding author

NeurIPS 2026

Abstract

We present PAI-Actor, a cinematic multi-character animation framework for character replacement in dynamic movie scenes. Unlike conventional animation systems that mainly drive a single static image or a single subject, our goal is to replace and animate multiple characters within real video clips while preserving the original scene dynamics, camera motion, and background content. This setting is particularly challenging because the generated characters must remain consistent with the source performance in motion and interaction, while also matching the surrounding background in lighting, shadow, composition, and overall cinematic appearance. To address this, we formulate multi-character animation as a structure-guided human recovery problem and build a movie-driven training pipeline from high-quality film data. Furthermore, to support practical cinematic production, we introduce a bidirectional-to-autoregressive distillation framework: we first train a bidirectional diffusion transformer for high-quality short-clip generation at 1080P resolution, and then distill it into an autoregressive video-to-video model for efficient inference and longer video generation. Experiments show that PAI-Actor enables high-fidelity multi-character animation with strong scene consistency, cinematic visual quality, and efficient long-form generation.

Feature Comparison

Feature	Wan-Animate	MultiAnimate	SCAIL	DOMO	Viggle	Kling O1	MoCha	Ours
1080P	✗	✗	✗	✗	✓	✓	✓	✓
Multi-Character	✗	✓	✓	✓	✓	✓	✗	✓
Long-Video Output (>10s)	✗	✗	✗	✓	✓	✓	✗	✓
Pose-Driven Control	✓	✓	✓	✓	✗	✓	✗	✓
Background Preservation	✓	✗	✗	✓	✓	✓	✓	✓
Occlusion Handling	✗	✗	✗	✗	✗	✗	✗	✓

Feature comparison between PAI-Actor and existing character-replacement approaches. PAI-Actor is the only method that jointly supports 1080P resolution, multi-character animation, long-form generation, pose-driven control, background preservation, and complex occlusion handling.

Method

Overview of the PAI-Actor framework. We cast multi-character animation as a structure-guided human recovery problem: original performers are masked out from movie clips, structured pose and face signals are rendered on the masked regions, and a diffusion transformer recovers the complete characters in the original video space. A bidirectional-to-autoregressive distillation strategy first trains a bidirectional teacher for 1080P short-clip quality, then distills it into an autoregressive video-to-video generator for efficient long-form synthesis.

Generation Results

Cinematic multi-character replacement results across diverse movie scenes. Generated characters preserve source-video motion, interaction, and camera dynamics while matching the surrounding scene lighting and composition.

Comparison with Baselines

Qualitative comparison against existing character-replacement systems. Baselines struggle with identity drift, background distortion, occlusion failures, and resolution degradation. PAI-Actor produces clean, identity-consistent, occlusion-aware replacements that preserve the original cinematic scene.

Ablation Study

Ablation on key components of PAI-Actor, including the structure-guided recovery formulation, movie-driven training data, and the bidirectional-to-autoregressive distillation. Each component contributes to final motion fidelity, identity preservation, and scene consistency.

More Animation Results

Additional multi-character animation results across diverse movie scenes. Each example compares PAI-Actor against baselines and the ground-truth video, for both single- and multi-character settings.

User Study

Human evaluation across 15 users covering Background Consistency, FG-BG Fusion, Illumination Harmony, and Character Consistency. PAI-Actor receives the highest vote share over open-source and commercial baselines in both single- and multi-character settings.

Quantitative Results

Character-Animation Subset

Quantitative comparison on the Character-Animation Subset under single-character (S) and multi-character (M) settings. The last four metrics are LLM-based perceptual scores. ^*Wan-Animate is applied one character at a time for the multi-character setting. Best per column in bold.

Character-Replacement Subset (LLM-based)

LLM-based evaluation (averaged GPT-5.1 & Gemini 3 Pro) on the Character-Replacement Subset, which has no ground truth and is scored perceptually. Best per column in bold.

Long-Video Subset

Long-Video results of the final causal (autoregressive) student on the Long-Video Subset. The distilled student sustains cinematic quality on long sequences that the bidirectional teacher cannot generate directly.

Ablation Study

Ablation on the Character-Animation Subset under single-character (S) and multi-character (M) settings. The final self-forcing causal student trades a small amount of reconstruction accuracy for autoregressive scalability. Best per column in bold.

Long-Form Generation (60s)

A continuous 60-second multi-character replacement result, demonstrating long-form temporal consistency and identity preservation without drift.