Zero4D: Training-Free 4D Video Generation From Single Video
Using Off-the-Shelf Video Diffusion Model

Abstract

Report issue for preceding element

Recently, multi-view or 4D video generation has emerged as a significant research topic. Nonetheless, recent approaches to 4D generation still struggle with fundamental limitations, as they primarily rely on harnessing multiple video diffusion models with additional training or compute-intensive training of a full 4D diffusion model with limited real-world 4D data and large computational costs. To address these challenges, here we propose the first training-free 4D video generation method that leverages the off-the-shelf video diffusion models to generate multi-view videos from a single input video. Our approach consists of two key steps: (1) By designating the edge frames in the spatio-temporal sampling grid as key frames, we first synthesize them using a video diffusion model, leveraging a depth-based warping technique for guidance. This approach ensures structural consistency across the generated frames, preserving spatial and temporal coherence. (2) We then interpolate the remaining frames using a video diffusion model, constructing a fully populated and temporally coherent sampling grid while preserving spatial and temporal consistency. Through this approach, we extend a single video into a multi-view video along novel camera trajectories while maintaining spatio-temporal consistency. Our method is training-free and fully utilizes an off-the-shelf video diffusion model, offering a practical and effective solution for multi-view video generation.

Report issue for preceding element

1 Introduction

Report issue for preceding element

Since the introduction of the diffusion and foundation models [9, 25, 40], 3D reconstruction has advanced significantly, leading to unprecedented progress in representing the real world in 3D models. Combined with generative models, this success drives a renaissance in 3D generation, enabling more diverse and realistic content creation. These advancements extend beyond static scene or object reconstruction and generation, evolving toward dynamic 3D reconstruction and generation that aims to capture the real world.

Report issue for preceding element

Prior works [4, 47, 27, 50, 2] leverage video diffusion models and Score Distillation Sampling (SDS) to enable dynamic 3D generation. However, most of these approaches primarily focus on generating dynamic objects in blank backgrounds (e.g., text-to-4D generation), leaving the challenge of reconstructing or generating real-world scenes from text condition or reference images or video. But, unlike the abundance of high-quality 3D and video datasets, extensive 4D datasets required for such 4D models remain scarce. Therefore, a fundamental challenge in training 4D generation models for real-world scenes is the lack of large-scale, multi-view synchronized video datasets.

Report issue for preceding element

To overcome these limitations, recent works such as 4DiM [37] proposes joint training diffusion model with 3D and video with scarce 4D dataset. CAT4D [39] proposes training multi-view video diffusion models by curating a diverse collection of synthetic 4D data, 3D datasets, and monocular video sources. DimensionX [28] trains the spatial-temporal diffusion model independently with multiple LoRA, achieved multi-view videos via an additional refinement process. Despite several approaches, the scarcity of high-quality 4D data makes it difficult to generalize to complex real-world scenes and poses fundamental challenges in training large multi-view video models.

Report issue for preceding element

To address these challenges, here we introduce a novel zero-shot framework called Zero4D, short for Zero-shot 4D video generator, which generates multi-view synchronized 4D video from single monocular video by leveraging off-the-shelf video diffusion model [5] without any training. Building upon the prior observations [31, 39] that 4D video is composed of multiple video frames arranged along the spatio-temporal sampling grid (i.e., camera view and time axes), generating a 4D video can be regarded as populating the sampling grid with consistent spatio-temporal frames. Consequently, our approach achieves this through two key steps: (1) We first designate the edge frames in the sampling grid as key frames and synthesize them. Specifically, we employ a depth-based warping technique as guidance in a video diffusion model, ensuring that the generated frames adhere to the underlying scene structure. (2) Inspired by ViBiDSampler [42], we then leverage the interpolation capabilities of a video model to fill in the remaining frames through bidirectional diffusion sampling, ensuring a fully populated and temporally coherent 4D grid. During these steps, our method imposes both spatial and temporal consistency across the grid. Our main contributions can be summarized as follows:

Report issue for preceding element

• We propose a novel framework that can generate 4D video from a single video via an off-the-shelf video diffusion model without any training or large-scale datasets. To the best of our knowledge, our approach is the first training-free method to generate synchronized multi-view video.

  Report issue for preceding element

• We introduce a synchronization mechanism to ensure high-quality generation while preserving global spatio-temporal consistency. This is achieved through alternating bidirectional video interpolation sampling along both the camera and temporal axes, effectively aligning motion and appearance across frames.

  Report issue for preceding element

• Our approach is computationally efficient, enabling high-quality multi-view video generation with minimal memory consumption, making it significantly more accessible than previous methods.

  Report issue for preceding element

2 Related Work

Report issue for preceding element

Figure 2: Reconstruction pipeline of Zero4D: (a) Key frame generation step: (a1) Given a fixed-viewpoint grayscale input video, we generate synchronized multi-view videos using the I2V diffusion model. (a2) We first synthesize key frames of the 4D grid through diffusion sampling, guided by warped views. (a3) Next, we generate the end-view video frames using a process similar to (a2), guided by warped views. (a4) Finally, we complete the rightmost column using diffusion-based interpolation sampling. (b) Spatio-temporal bidirectional interpolation step: Starting from the initial noise in (a4), we denoise the remaining frames through a camera-axis interpolation denoising step. At this stage, the noisy frame $x_{t}[:,i]$ is updated based on edge frames along the camera axis. Then, the time-axis interpolation step follows, where perturbed frames are denoised through interpolation along the time axis of the 4D grid, conditioned on edge frames in the time axis. The spatio-temporal bidirectional sampling alternates between camera-axis interpolation and time-axis interpolation, progressively refining noisy latents into clean frames. Through this process, we obtain globally coherent spatio-temporal 4D grid videos.

Report issue for preceding element

Dynamic 3D reconstruction. Numerous studies have explored methods for representing the dynamic nature of the real world in 3D models. DyNeRF [18] introduced 4D novel view synthesis using multi-view video inputs. Building on this foundation, subsequent research has focused on 4D reconstruction using posed multi-view inputs [38, 43, 12, 44]. However, all these methods require a large number of multi-view images with known camera parameters, which poses a significant challenge. With the success of Dust3R [35], which learns 3D representations in a self-supervised manner from unposed images without requiring camera parameters, many subsequent models have attempted to reconstruct real-world dynamic scenes based on this foundational approach. Monst3r [49], built on DUSt3R, enables the reconstruction of both dynamic point clouds and camera poses from unposed single-frame video inputs. Shape-of-Motion [33] attempts 4D reconstruction by tracking points from dynamic motion in single-video inputs without camera parameters. CUT3R [34] reconstructs dynamic scenes from single-video inputs by leveraging state updates of visual tokens within a ViT encoder.

Report issue for preceding element

Video generation with camera control.

Report issue for preceding element

Several studies try to train a multi-view diffusion model for spatially consistent image generation [26, 32, 20, 14, 7, 22]. Camco [41] fintunes pre-trained video diffusion model with injecting Plücker embedding vector into the specific layer in the model. ReCapture [48] trains the novel camera trajectory video diffusion model from a single reference video with existing scene motion. They leverage trainable multiple LoRA layers on a video model with a camera parameter label to regenerate the anchor video into a natural and temporally consistent novel view video. AC3D [3] enhances Video DiT [10] with ControlNet based camera conditioning, keeping the large frozen VDiT backbone for video synthesis while using lightweight modules to inject camera information. Cameractrl [8] proposes a plug-and-play camera module in the video diffusion model to control video generation with precise and smooth camera view points.

Report issue for preceding element

4D generation. Recent advancements in text-to-4D generation have been driven by numerous pioneering works exploring various conditioning methods. Among these, several approaches have leveraged score distillation sampling in conjunction with video diffusion models or multi-view image diffusion models to generate 4D content from text prompts [4, 47, 27, 50, 2]. However, these approaches largely focus on generating dynamic objects in blank backgrounds. Generating full 4D scenes under given constraints has recently emerged as an important research direction with limited prior work. A notable example is [39], which synthesizes 4D videos conditioned on multiple input modalities using a multi-view video model. They extend an image diffusion model into a multi-view video diffusion framework, trained on a carefully curated spatio-temporal dataset. To ensure consistency across viewpoints, they employ an alternating sampling strategy. Similarly, [30] introduces a framework for novel view synthesis of dynamic 4D scenes from a single video. This method is trained on synthetic multi-view video data with corresponding camera poses, enabling high-fidelity 4D reconstructions. Concurrently, [46] propose text-to-4D scene generation pipelines that integrate video diffusion models with canonical 3D Gaussian Splatting (3DGS) [17], ensuring spatio-temporal consistency in the generated 4D outputs. Furthermore, [31] enhances video diffusion models by introducing a parallel camera-temporal token stream and a learnable synchronization layer, which effectively fuses independent tokens to maintain camera and temporal consistency across generated frames.

Report issue for preceding element

3 Zero4D

Report issue for preceding element

Let $x[i,j]\in\mathbb{R}^{H\times W},i=1,\cdots,N,j=1,\cdots,F$ denotes the image at the $i$-th camera viewpoint and the $j$-th temporal frame, where $H$ and $W$ denotes the height and width of the image, respectively (see Fig. 2(a)). Then, the input video captured from a single camera viewpoint $c$ is denoted as $x[c,:]$, whereas the multi-view images at the temporal frame $f$ is represented by $x[:,f]$. Then, The goal of Zero4D is to populate the spatio-temporal sampling grid (or camera-time grid) $x[:,:]$ by generating frames across multiple camera poses.

Report issue for preceding element

As illustrated in Fig. 2, the overall reconstruction pipeline of Zero4D is composed of two steps: 1) key frame generation and 2) Spatio-temporal bidirectional interpolation along the time and camera axis in an alternating manner. In this section, we describe each in detail.

Report issue for preceding element

3.1 Key Frame Generation

Report issue for preceding element

As shown in Fig. 2(a), the key frame generation is achieved through three successive steps. Specifically, given a fixed-viewpoint grayscale input video denoted by $x[1,:]$, we first perform novel view synthesis, which is followed by end-view video frame generation. These two steps are achieved through diffusion sampling, guided by warped views. Finally, we complete the rightmost column using diffusion-based interpolation sampling.

Report issue for preceding element

Novel view synthesis (a2). At the first stage of key frame generation, we synthesize a novel view $x[:,1]$ from the first frame $x[1,1]$ using the I2V diffusion model. We incorporate the warped frames $x_{w}[:,1]$ as guidance to ensure the generated novel views align with the warped images from input video warping.

Report issue for preceding element

The warped frames $x_{w}[:,:]$ is computed as follows. Given an input video $x[1,:]$, we generate novel views by first estimating a per-frame depth map $D[1,:]$ using a monocular depth estimation model [23]. This depth information enables depth-based geometric warping, where each frame of the input video is unprojected into 3D space and reprojected into a target viewpoint in $p(n)\in\mathcal{P}_{N}$ where $\mathcal{P}_{N}$ defines the desired set of camera views. This produces the warped frames:

Report issue for preceding element

$$x_{w}[n,:]=\mathcal{W}(x[1,:],D[1,:],p(n),K),$$ (1)

for $n=1,\cdots,N$, where $K$ is the intrinsic camera matrix. The warping function $\mathcal{W}(\cdot)$ unprojects each pixel using its estimated depth and reprojects it into the target view. Formally, for each pixel location $r_{i}$ in the $i$-view, the warped pixel location $r_{j}$ in the novel view at the $j$-th camera location is computed as:

Report issue for preceding element

$$r_{j}=KP_{i\to j}D_{i}(r_{i})K^{-1}r_{i},$$ (2)

where $P_{i\to j}$ is the transformation from the input to the novel view, and $D_{i}(r_{i})$ is the depth at $r_{i}$. Since $r_{j}$ may not align exactly with integer pixel locations, interpolation is applied to assign pixel values.

Report issue for preceding element

However, missing regions (e.g., occlusions from depth-based projection) often appear in $x_{w}$. To address this, we utilize a video diffusion model parameterized by $\theta$ to inpaint the missing regions and ensure consistency within the 4D video grid. This can be considered as conditional sampling under the condition of the warped image, occlusion mask and the input video conditioning. For the case of novel view synthesis at the temporal frame index $j=1$, this corresponds to

Report issue for preceding element

$$x[:,1]\sim p_{\theta}(x[:,1]\mid x_{w}[:,1],m_{w}[:,1],c[1,1]),$$ (3)

where $p_{\theta}$ corresponds to the conditional distribution from the trained diffusion model, $m_{w}[:,:]$ is an occlusion mask that identifies missing pixels, and $c[1,1]$ provides the conditioning input from $x[1,1]$. The specific details of conditional video diffusion sampling will be described in Section 3.3.

Report issue for preceding element

Algorithm 1 $I_{\theta}:$ A sampling step of extended ViBiDSampler for bidirectional interpolation.

1:function $I_{\theta}$($x_{t},\sigma_{t},c_{start},c_{end},x_{w}$)

2:     ${\color[rgb]{1,.5,0}\hat{x}_{c_{start}}}\leftarrow D_{\theta}(x_{t};\sigma_{t},c_{start})$ $\triangleright$ EDM denosing

3:     ${\color[rgb]{1,0,0}\bar{x}_{c_{start}}}\leftarrow{\color[rgb]{1,.5,0}\hat{x}_{c_{start}}}\cdot{m}+{\color[rgb]{1,0,0}{x}_{w}}\cdot(1-m)$

4:     $x_{t-1,c_{start}}\leftarrow{\color[rgb]{1,0,0}\bar{x}_{c_{start}}}+\frac{\sigma_{t-1}}{\sigma_{t}}(x_{t}-\hat{x}_{\emptyset})$

5:     $x_{t},c_{start}\leftarrow x_{t-1,c_{start}}+\sqrt{\sigma_{t}^{2}-\sigma_{t-1}^{2}}\epsilon$ $\triangleright$ Re-noise

6:     $x_{t},c_{start}\leftarrow\text{flip}(x_{t},c_{start})$ $\triangleright$ Time reverse

7:     $\hat{x}^{\prime}_{c_{end}}\leftarrow D_{\theta}(x^{\prime}_{t},c_{start};\sigma_{t},c_{end})$ $\triangleright$ EDM denoising

8:     ${\color[rgb]{1,0,0}\bar{x}^{\prime}_{c_{end}}}\leftarrow{\color[rgb]{1,.5,0}\bar{x}^{\prime}_{c_{end}}}\cdot{m}+{\color[rgb]{1,0,0}{x}_{w}}\cdot(1-m)$

9:     $x^{\prime}_{t-1}\leftarrow{\color[rgb]{1,0,0}\bar{x}^{\prime}_{c_{end}}}+\frac{\sigma_{t-1}}{\sigma_{t}}(x^{\prime}_{t},c_{start}-\hat{x}^{\prime}_{\emptyset})$

10:     $x^{\prime}_{t-1}\leftarrow\text{flip}(x^{\prime}_{t-1})$ $\triangleright$ Time reverse

11:     return $x_{t-1}$

12:end function

Report issue for preceding element

End view video generation (a3). Similarly, we can synthesize the end view video $x[N,:]$ from the generated view $x[N,1]$ utilizing warp-guided diffusion sampling.

Report issue for preceding element

$$x[N,:]\sim p_{\theta}(x[N,:]\mid x_{w}[N,:],m_{w}[N,:],c[N,1]).$$ (4)

This process follows the same video sampling approach as first-frame novel view synthesis; however, it differs in that it synthesizes the video from the final camera position.

Report issue for preceding element

End frame novel view synthesis (a4). Finally, we generate video at the end-frame novel view $x[:,F]$, which constitutes the rightmost column of the 4D grid in Fig. 2(a). Given that we already have $x[1,F]$ from the input video and the synthesized end-view frame $x[N,F]$ derived from $x[N,:]$, we incorporate both images to enhance consistency. To this end, we employ the idea of ViBiDSampler [42] - the state-of-the-art video interpolation method that allows for the simultaneous conditioning on both $c[1,F]$ and $c[N,F]$. As described in Algorithm 1, the main innovation from ViBiDSampler is that we additionally incorporate the conditions from the warped image and its mask. Accordingly, we synthesize the last column $x[:,F]$ using bidirectional video diffusion sampling:

Report issue for preceding element

$$x_{t}[:,F] =I_{\theta}\big{(}x_{t}[:,F],\sigma_{t},c[0,F],c[N,F],x_{w}[:,F]\big{)}$$ (5)

$$\quad\text{for}\quad t=T\to 0.$$

where $I_{\theta}$ denotes the one-step bidirectional video interpolation using the extended version of ViBiDSampler. The final novel-view frame $x[:,F]$ is obtained iteratively by applying $I_{\theta}$ over diffusion time steps $t=T\to 0$

Report issue for preceding element

3.2 Spatio-Temporal Bidirectional Interpolation

Report issue for preceding element

As shown in Fig. 2(b), once the keyframes are generated, the remaining task is to fill in the missing sampling grid at the center so the final resulting 4D video remains consistent across both the camera and time axes. Accordingly, it is essential to perform conditioned sampling using the key frames and adjacent frames from the camera and temporal axes. However, a naive image-to-video diffusion model can only condition on a single or two end frames.

Report issue for preceding element

To address this challenge, inspired by ViBiDSampler [42], we propose a novel Spatio-Temporal Bidirectional Interpolation (STBI), which enables simultaneous conditioning on both the camera and time axes. The key idea is alternating the aforementioned bidirectional sampling along both camera and time axes so that the overall diffusion sampling trajectory is driven to satisfy multiple conditions from the key frames. More details follow.

Report issue for preceding element

Camera axis interpolation. Starting from the initial noise $x_{T}[:,:]\sim\mathcal{N}(0,I)$ (line 9 in algorithm 2), we then select a specific frame in the 4D grid (a column) $x_{t}[:,i]$, and perform an interpolation denoising process using the edge-frame conditions $c[1,i]$ and $c[N,i]$:

Report issue for preceding element

$$x_{t-1}[:,i]\leftarrow I_{\theta}(x_{t}[:,i],\sigma_{t},c[1,i],c[N,i],x_{w}[:,i])$$ (6)

In this process, the image condition $c[0,i]$ is applied first, along with the warped view to guide the diffusion denoising step. The video is then perturbed with noise again, flipped along the camera axis, and subjected to another diffusion denoising step using $c[N,i]$ as the condition. Through these two conditioning steps, $x_{t}[:,i]$ integrates information from both $c[1,i]$ and $c[N,i]$, enabling interpolation-based denoising that preserves consistency across the camera axis. Before proceeding with time axis interpolation, we apply a re-noising step, to ensure smooth transitions across generated frames.

Report issue for preceding element

Time axis interpolation. After ensuring spatial consistency across the camera axis, we interpolate frames along the time axis to maintain temporal coherence. For each row $x_{t}[j,:]$ in the 4D grid, we perform an interpolation denoising step (7) using the start and end frame conditions $c[j,1]$ and $c[j,F]$.

Report issue for preceding element

$$x_{t}[j,:]\leftarrow I_{\theta}\bigl{(}x_{t}[j,:],\sigma_{t},c[j,1],c[j,F],x_{w}[j,w]\bigr{)}$$ (7)

Initially, $c[j,1]$ is applied along with the warped view to guide the diffusion denoising step. The frame is then perturbed with noise, flipped along the time axis, and another diffusion denoising step is performed using $c[j,F]$ as the condition. Through this bidirectional conditioning process, $x_{t}[j,:]$ effectively integrates information from both $c[j,1]$ and $c[j,F]$, facilitating interpolation-based denoising that ensures smooth transitions along the time axis.

Report issue for preceding element

Throughout the diffusion step, we perform denoising by alternating interpolation along the camera axis and time axis. This approach maintains global coherence while ensuring consistency in multi-view video generation. The overall sampling procedure of Zero4D is summarized in Table 2 in Appendix Section 6.

Report issue for preceding element

3.3 Details of Conditional Video Diffusion

Report issue for preceding element

In this work, we build upon Stable Video Diffusion (SVD) [5], an image-to-video diffusion model that follows the principles of the EDM framework [15]. SVD utilizes an iterative denoising approach based on an Euler step method, which progressively transforms a Gaussian noise sample $x_{T}$ into a clean signal $x_{0}$:

Report issue for preceding element

$$x_{t-1}(x_{t};\sigma_{t},c):=\hat{x}_{c}(x_{t})+\frac{\sigma_{t-1}}{\sigma_{t}}\,\bigl{(}x_{t}-\hat{x}_{c}(x_{t})\bigr{)},$$ (8)

where the initial noise is $x_{T}\sim\mathcal{N}(0,I)$, $\hat{x}_{c}(x_{t})$ is the denoised estimate by Tweedie’s formula using the score function trained by the neural network parameterized by $\theta$, and $\sigma_{t}$ is the discretized noise level for each timestep $t\in[0,T]$.

Report issue for preceding element

Figure 3: Result from zero-4D. Our model can generate high-quality multi-view videos from a single input video. By rotating the viewpoint in 15-degree increments, we produce novel-view videos based on the given input. As shown in the figure, the generated videos maintain consistency across multiple views and frames, effectively synthesizing perspectives that were not visible in the original video.

Report issue for preceding element

Now, we describe how to modify SVD to enable conditional sampling under the condition on the warped image $x_{w}$, the occlusion mask $m$ and the conditioning input $c$. For convenience, we refer to $x_{t}[:,:]$ as $x_{t}$. From the formulation of the reverse diffusion sampling process in (8), The reverse diffusion process can be modulated by conditioning on a known scene-prior $x_{\text{known}}$, as proposed in [21]:

Report issue for preceding element

$$\bar{x}_{c}(x_{t})=\hat{x}_{c}(x_{t})\cdot{m}+{x_{\text{known}}}\cdot(1-m),$$ (9)

where $m$ is a mask that determines which parts of the scene are known, guiding the denoising process by preserving the known regions while allowing the diffusion model to inpaint the missing areas. In our approach, rather than relying on an externally defined scene-prior $x_{\text{known}}$, we leverage the warped frames $x_{w}$ obtained from depth-based warping as the conditional guidance. Specifically, we redefine the denoising process by replacing $x_{\text{known}}$ with $x_{w}$ and substituting $m$ with the occlusion mask $m_{w}$:

Report issue for preceding element

$$\bar{x}_{c}(x_{t})=\hat{x}_{c}(x_{t})\cdot{m_{w}}+{x_{w}}\cdot(1-m_{w}).$$ (10)

Here, the occlusion mask $m_{w}$ ensures that the visible regions in $x_{w}$ directly guide the denoising process, while the unseen parts are inpainted using the learned prior. By incorporating this modified formulation into the reverse diffusion process, we obtain the following sampling update:

Report issue for preceding element

$$x_{t-1}(x_{t};\sigma_{t},c)\leftarrow\bar{x}_{c}(x_{t})+\frac{\sigma_{t-1}}{\sigma_{t}}\,\bigl{(}x_{t}-\hat{x}_{c}(x_{t})\bigr{)},$$ (11)

where the target camera viewpoints influence the generated frames through the depth-warped observations $x_{w}$, ensuring geometric consistency during video synthesis. Throughout the diffusion reverse sampling process, we iteratively apply this procedure. Additionally, following the approach described in [21, 19], we incorporate resampling annealing to further enhance the output quality.

Report issue for preceding element

4 Experiments

Report issue for preceding element

We used the SVD[5] as an I2V model for all experiments without any training. The image resolution was fixed at 576×1024, with 25 cameras and a sequence length of 25 frames, a total of multi-view video frames are 625=$25^{2}$. All frames were generated to form a multi-view video following the target camera trajectory. For depth-based warping, we utilized off-the-shelf depth models [23, 11]. For depth-based warping, we utilized off-the-shelf depth models [23, 11] with a view variation of 15 degrees.

Report issue for preceding element

Figure 4: Baseline comparison. We compare our approach with existing multi-view video generation methods, including SV4D [40] and GCD [29]. These methods take a single-view video as input and generate a novel-view video. Since SV4D and GCD rely on training-based models, their performance degrades when processing videos that deviate from the distribution of their training data, often resulting in blurry or distorted outputs. In contrast, our method operates without additional training, effectively preserving spatio-temporal consistency and producing high-quality multi-view videos.

Report issue for preceding element

Method	FVD $\downarrow$	FID $\downarrow$	LPIPS $\downarrow$	Subject Consistency $\uparrow$	Background Consistency $\uparrow$	Temporal Flickering $\uparrow$	Motion Smoothness $\uparrow$	Dynamic Degree $\uparrow$	Image Quality $\uparrow$	Aesthetic Quality $\uparrow$
SV4D [40]	1502.4	200.74	0.4002	86.46%	92.71%	92.04%	97.78%	55.00%	97.78%	50.36%
CGD [36]	1584.4	238.17	0.4947	91.73%	95.55%	98.41%	98.86%	20.00%	44.55%	42.67%
Ours	896.29	135.08	0.3997	94.18%	94.53%	97.35%	98.13%	55.77%	50.08%	62.09%

Table 1: Quantitative result. We evaluate our method against SV4D and GCD on FVD, FID, LPIPS, and VBench[13], comparing multi-view video results based on novel view generation from a fixed camera viewpoint. Our method achieves the best performance in both frame consistency across videos and image quality of individual frames.

Report issue for preceding element

Baseline Models

Report issue for preceding element

We selected publicly available methods as baselines. SV4D [40] is an image-to-video (I2V) model that generates multiple novel-view videos from a single input, constructing a 4D image matrix of the object. It synthesizes target-view videos by applying consistent azimuthal displacement, producing 21-frame outputs at $512\times 512$ resolution. GCD [29] also takes a single video as input, allowing the selection of specific azimuth and elevation angles to generate novel views of dynamic 4D scenes. It produces 25-frame videos at $384\times 256$ resolution. All experiments were conducted on an RTX 4090 GPU (24GB VRAM). While both SV4D and GCD require extensive training, which is impractical on a single GPU, our approach eliminates the need for training, enabling efficient execution on a single RTX 4090.

Report issue for preceding element

Figure 5: Ablation study. Given a single input video, our model generates high-quality multi-view videos. When multi-view video generation is performed without Spatio-Temporal Bidirectional Interpolation (STBI), each frame is synthesized independently, leading to inconsistencies across frames (see the red box). In contrast, STBI aggregates global information, enabling the generation of multi-view videos with improved frame consistency.

Report issue for preceding element

Multi-View Video

Report issue for preceding element

To validate our method, we conducted experiments on 10 DAVIS [24] and 20 Pexel scenes. Novel views were synthesized by adjusting the target camera’s azimuth in 15-degree increments, following the evaluation protocol in [29]. For fair comparison, videos from our method and SV4D were resized to $384\times 256$. To assess video quality, we utilized VBench [13], which evaluates seven key aspects of video generation, including subject identity retention, motion coherence, and temporal consistency. It incorporates advanced feature extractors such as DINO [6] for consistency and MUSIQ [16] for image quality, offering a detailed performance analysis. We further compared the ability of each baseline model to generate novel view videos given a single input video. As shown in Figure 4, our method demonstrates robust novel view synthesis, whereas SV4D and GCD exhibit reduced object consistency and distorted motion. This observation is further supported by the quantitative results in Table 1, where our method outperforms others in FVD, FID, LPIPS, and Subject Consistency, demonstrating superior overall video generation quality.

Report issue for preceding element

Bullet-Time and Camera Control

Report issue for preceding element

We evaluate the generated multi-view video by controlling the camera at fixed time points within the video. To further assess 3D consistency along the camera axis at fixed time, we utilize MEt3R [1], a state-of-the-art 3D reconstruction model. This measures 3D consistency in videos based on unposed images using DUSt3R[35] as its backbone. Currently, no open-source multi-view video generation model explicitly supports bullet-time generation. Thus, we conducted an ablation study to analyze its impact. To comprehensively evaluate bullet-time video quality, we employed FVD, FID, and LPIPS. By incorporating both spatio-temporal bidirectional interpolation (STBI) and warping, we achieved well-synchronized multi-view videos across both camera and time axes. As shown in Table 3, our method outperforms others across all key metrics. However, when STBI was not used, each frame underwent novel view synthesis independently. While this preserved per-frame 3D consistency, the absence of temporal coherence led to inferior FVD performance compared to our full method, which maintains global consistency across the entire video.

Report issue for preceding element

Method	FVD $\downarrow$	FID $\downarrow$	LPIPS $\downarrow$	Subject Consistency $\uparrow$	Background Consistency $\uparrow$	Temporal Flickering $\uparrow$	Motion Smoothness $\uparrow$	Dynamic Degree $\uparrow$	Image Quality $\uparrow$	Aesthetic Quality $\uparrow$
Ours	896.29	135.08	0.3997	94.18%	94.53%	97.35%	98.13%	55.77%	50.08%	62.09%
w/o STBI	1419.8	108.92	0.3797	92.57%	93.59%	94.49%	94.49%	100%	52.93%	67.57%
w/o warp	1830.5	205.08	0.4547	92.91%	93.45%	97.48%	98.01%	48.28%	44.97%	47.26%

Table 2: Fixed view video Quantitative ablation. We performed ablation studies on videos synthesized from fixed novel viewpoints, showing that our method achieves optimal performance when all components are incorporated.

Report issue for preceding element

Method	FVD $\downarrow$	FID $\downarrow$	LPIPS $\downarrow$	MEt3R $\downarrow$
Ours	1470	106.31	0.3174	0.039
w/o STBI	1607	108.21	0.3181	0.031
w/o WAP	1923	179.99	0.3617	0.082

Table 3: Bullet-time Video Quantitative Ablation. We conducted ablation experiments to evaluate how well videos generated at fixed time points maintain quality. The results confirm that our method achieves the best performance when fully utilized.

Report issue for preceding element

Ablation

Report issue for preceding element

We conducted comparative experiments under the following conditions: (1) Without warped frame guidance (see (10)). In this setting, we excluded key frame generation and removed warped frames from the input during camera-time interpolation. As shown in Table 2, conditioning the diffusion model solely on keyframes results in less distinct and less detailed images. This effect is quantitatively evident in the table, where the absence of warped frame guidance leads to degraded performance across key metrics. (2) Without spatio-temporal bidirectional interpolation (STBI). We also examined cases where STBI was omitted, generating each novel view independently per frame. Without global aggregation along the time axis, this approach fails to maintain multi-view coherence. In contrast, our method incorporates key frame generation to capture global scene information, enabling bidirectional frame synthesis. As reflected in Table 3, this results in better synchronization and improved temporal consistency, highlighting the role of STBI in preserving video coherence.

Report issue for preceding element

User Study

Report issue for preceding element

Method	View Angle	General Quality	Smoothness	BG Quality
Ours	67%	70%	60%	68%
GCD	22%	21%	27%	22%
SV4D	11%	9%	13%	10%

Table 4: User study. Winning rates across four key evaluation metrics. Our method consistently outperforms the baselines, particularly in General Quality and Background Quality.

Report issue for preceding element

To evaluate our approach, we conducted a user study comparing Ours, GCD, and SV4D across four key metrics: View Angle, General Quality, Smoothness, and Background Quality. Participants viewed generated videos and selected the most visually appealing results. As shown in Table 4, our method consistently achieved the highest user preference, particularly in General Quality (70%) and Background Quality (68%), highlighting superior fidelity and scene preservation. View Angle (67%) results confirm precise novel view synthesis, while Smoothness (60%) indicates seamless transitions with minimal distortion.

Report issue for preceding element

5 Conclusion

Report issue for preceding element

In this work, we introduced a novel training-free approach for synchronized multi-view video generation using an off-the-shelf video diffusion model. Our method generates high-quality 4D video grids through depth-based warping and spatio-time bidirectional interpolation, ensuring structural consistency across spatial and temporal domains. Unlike prior methods requiring extensive training on large-scale 4D datasets, our framework achieves competitive performance without any training. Extensive experiments show that our approach generates synchronized multi-view videos with superior subject consistency, motion smoothness, and temporal stability. It outperforms baselines in aesthetic quality and imaging fidelity while remaining computationally efficient. Our framework offers a practical solution for multi-view video generation, especially when large-scale 4D datasets and computational resources are scarce. Future work may explore dynamic scenes or adaptive interpolation strategies to enhance novel view synthesis.

Report issue for preceding element

Supplementary Material

Report issue for preceding element

6 Pseudocode implementation of Zero4D

Report issue for preceding element

Algorithm 2 Zero4D overall pipeline

1:$x[1,:],x_{w}[:,:],m_{w}[:,:]$

2:$x[:,1]\sim p_{\theta}(x[:,1]\mid x_{w}[:,1],m_{w}[:,1],c[1,1]).$

3:$x[N,:]\sim p_{\theta}(x[N,:]\mid x_{w}[N,:],m_{w}[N,:],c[N,1]).$

4:$x_{t}[:,F]\sim\mathcal{N}(0,I)$

5:for $t=T\textbf{ to }1$ do

6:    $x_{t}[:,F]\leftarrow I_{\theta}(x_{t}[:,F],\sigma_{t},c[N,1],c[N,F],x_{w}[:,F])$

7:end for$\triangleright$ Finishing Keyframe generation

8:$c[1,:],\,c[:,1]\;\;\leftarrow\;\text{Encode}\bigl{(}x[1,:],\,x[:,1]\bigr{)}$

9:$c[:,F],\,c[N,:]\;\;\leftarrow\;\text{Encode}\bigl{(}x[:,F],\,x[N,:]\bigr{)}$

10:$x_{T}[:,:]\sim\mathcal{N}(0,I)$

11:for $t=T$ to $1$ do$\triangleright$ Camera-time BiDi interpolation

12:    for $i=1$ to $n$ do$\triangleright$ Camera axis interpolation

13:         $x_{t-1}[:,i]\leftarrow I_{\theta}(x_{t}[:,i],\sigma_{t},c[1,i],c[N,i],x_{w}[:,i])$

14:         $x_{t}[{:,i}]\leftarrow x_{t-1}[:,i]+\sqrt{\sigma_{t}^{2}-\sigma_{t-1}^{2}}\epsilon$ $\triangleright$ Renoise

15:    end for

16:    for $j=1$ to $m$ do$\triangleright$ Time axis interpolation

17:         $x_{t}[j,:]\leftarrow I_{\theta}(x_{t}[j,:],\sigma_{t},c[j,1],c[j,F],x_{w}[j,:])$

18:    end for

19:end for

20:return $x_{0}[:,:]$

Report issue for preceding element

The following algorithm outlines the process of generating multi-view videos from a single monocular video. We begin by applying novel view synthesis to the first frame of the input video, utilizing an I2V diffusion model[5] to generate the novel view frame $x[:,1]$. To guide this process, we incorporate warping-based priors from the original video, enabling inpainting-based synthesis. Using an off-the-shelf depth estimation model [23], we warp the original frame to novel viewpoints, as illustrated in Figure 6. As shown in the figure, occluded regions resulting from the warp operation appear black, allowing us to extract an opacity mask. Inspired by [21, 45, 19], we adopt a mask inpainting approach, where inpainting is performed on the estimated noisy frame $\hat{x}_{0}[:,1]$. Rather than applying inpainting at every denoising step, as in [19], we utilize a re-noising process within the diffusion model’s denoising step to refine the final synthesis by reducing artifacts and enhancing structural coherence. Once $x[:,1]$ is generated, we apply the same inpainting and re-noising strategy to synthesize the novel-view frame $x[N,:]$. Furthermore, during the bidirectional interpolation process, we incorporate an additional re-noising step to enhance temporal and spatial consistency.

Report issue for preceding element

Algorithm 3 Novel view synthesis method from [19]

Input: warped frames $x_{w}$, opacity mask $\mathbf{m}$
Output: input video $x_{0}$

Report issue for preceding element

1:$x_{T}\sim\mathcal{N}(0,1)$

2:for $t=T,\dots,1$ do

3:       if $t>T-T^{\text{guide}}$ then

4:             for $r=1,\dots,R$ do

5:                    $\hat{x}_{0}=\text{Predict}(x_{t})$

6:                    if $r\leq R^{\text{guide}}$ then

7:                          ${\color[rgb]{1,.5,0}\hat{x}_{0}}\leftarrow D_{\theta}(x_{t};\sigma_{t},c_{x_{0}})$

8:                          ${\color[rgb]{1,0,0}\bar{x}_{0}}\leftarrow{\color[rgb]{1,.5,0}\hat{x}_{0}}\cdot{m}+{\color[rgb]{1,0,0}{x}_{w}}\cdot(1-m)$

9:                    else

10:                          ${\color[rgb]{1,0,0}\bar{x}_{0}}={\color[rgb]{1,.5,0}\hat{x}_{0}}$

11:                    end if

12:                    $x_{t-1}\leftarrow{\color[rgb]{1,0,0}\bar{x}_{0}}+\frac{\sigma_{t-1}}{\sigma_{t}}(x_{t}-\hat{x}_{0})$

13:                    if $r<R$ then

14:                          $x_{t}\sim\mathcal{N}(\bar{x}_{0}^{,}\sigma_{t})$

15:                    end if

16:             end for

17:       else

18:             ${\hat{x}_{t-1}}\leftarrow D_{\theta}(x_{t};\sigma_{t},c_{x_{0}})$

19:             $x_{t-1}\leftarrow{\color[rgb]{1,0,0}\bar{x}_{0}}+\frac{\sigma_{t-1}}{\sigma_{t}}(x_{t}-\hat{x}_{0})$

20:       end if

21:end for

22:return $x_{0}$

Report issue for preceding element

Figure 6: Input Video Warping. Given a single video, we utilize an off-the-shelf depth estimation model to generate warped frames from novel viewpoints.

Report issue for preceding element