Progressive Autoregressive Video Diffusion Models

Desai Xie1,2, Zhan Xu2, Yicong Hong2, Hao Tan2 Difan Liu2, Feng Liu2, Arie Kaufman1, Yang Zhou2,
1Stony Brook University    2Adobe Research   


Autoregressive long video generation at 60 seconds with progressive noise levels.
arxiv Code

Abstract

Current frontier video diffusion models have demonstrated remarkable results at generating high-quality videos. However, they can only generate short video clips, normally around 10 seconds or 240 frames, due to computation limitations during training. In this work, we show that existing models can be naturally extended to autoregressive video diffusion models without changing the architectures. Our key idea is to assign the latent frames with progressively increasing noise levels rather than a single noise level, which allows for fine-grained condition among the latents and large overlaps between the attention windows. Such progressive video denoising allows our models to autoregressively generate video frames without quality degradation or abrupt scene changes. We present state-of-the-art results on long video generation at 1 minute (1440 frames at 24 FPS).

Progressive Autoregressive Video Diffusion Models

Current frontier video diffusion models can only generate short video clips (e.g. 10 seconds or 240 frames) due to GPU memory constraints during training. To enable long video generation, autoregressively applying video diffusion models is the straightforward solution.


Abilities of teacher forcing, full-sequence diffusion, and Diffusion Forcing.

Comparison of autoregressively applying video diffusion models with replacement methods (left) vs. our progressive autoregressive video diffusion models (right).

Existing methods for autoregressively applying video diffusion models, i.e. the replacement methods, directly replace the noisy latent frames with the condition latents and then denoise the latents like regular video diffusion models.

Our models denoises the latent frames progressively, where each subsequent latent has a slightly higher noise level than the previous one. This provides more fine-grained condition, where the later frames with more uncertainty can follow the pattern of the earlier, more certain frames. Additionally, the attention windows of our models can have larger overlaps without extra computation cost. As a result, our models can autoregressively generate video frames without quality degradation or abrupt scene changes.

Results

We implement our progressive autoregressive generation and two baseline methods, replacement-with-noise and replacement-without-noise, on two pre-trained video diffusion models, Open-Sora v1.2 and a modified variant of Open-Sora, denoted as M. We also compare to Stable Video Diffusion (SVD-XT), which does I2V generation iteratively and StreamingSVD from StreamingT2V, which trains additional modules for autoregressive video generation.


Our Results:

Progressive-Autoregressive with M



Comparison with baselines

Progressive-Autoregressive with M (ours)

Replacement-with-noise with M

Progressive-Autoregressive with Open-Sora (ours)

Replacement-without-noise with Open-Sora

StreamingSVD from StreamingT2V

SVD-XT from Stable Video Diffusion