FileNotFoundError

[Ysz2022]

When I want to generate 4D content from a monocular video, I met this error:

File "/userhome/4Diffusion-master/threestudio/models/prompt_processors/base.py", line 346, in configure
self.load_text_embeddings()
File "/userhome/4Diffusion-master/threestudio/models/prompt_processors/base.py", line 402, in load_text_embeddings
self.text_embeddings = self.load_from_cache(self.prompt)[None, ...]
File "/userhome/4Diffusion-master/threestudio/models/prompt_processors/base.py", line 420, in load_from_cache
raise FileNotFoundError(
FileNotFoundError: Text embedding file .threestudio_cache/text_embeddings/2c4b12d35cd3ecaf7c28def82abcd4cf.pt for model stabilityai/stable-diffusion-2-1-base and prompt [baby yoda in the style of Mormookiee] not found.

Do you know why this error happened and how can I fix it ?

[aejion]

Thanks for your attention!
The prompt processor automatically generates the text embedding file. If the file is not found, please delete the '.threestudio_cache' folder and re-run our code.

[Ysz2022]

I wonder how much GPU memory is required for training the 4D-NeRF. I ran this comment on 4 NVIDIA 4090 GPUs but met Out-Of-Memory error:
python launch.py --config ./configs/4diffusion.yaml --train \
system.prompt_processor_multi_view.prompt='baby yoda in the style of Mormookiee'
system.prompt_processor_multi_view.image_path='./threestudio/models/imagedream/assets/yoda/0_rgba.png'
data.multi_view.image_path='./threestudio/models/imagedream/4dm/yoda'
system.prompt_processor_multi_view.image_num=8
system.prompt_processor_multi_view.total_num=25
data.multi_view.anchor_view_num=0

The OOM error is reported that:
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.01 GiB (GPU 0; 23.65 GiB total capacity; 20.81 GiB already allocated; 290.81 MiB free; 22.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.01 GiB (GPU 1; 23.65 GiB total capacity; 20.81 GiB already allocated; 290.81 MiB free; 22.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.01 GiB (GPU 2; 23.65 GiB total capacity; 20.81 GiB already allocated; 290.81 MiB free; 22.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.01 GiB (GPU 3; 23.65 GiB total capacity; 20.81 GiB already allocated; 290.81 MiB free; 22.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Epoch 0: | | 0/? [00:00<?, ?it/s]

It seems that even using 4 4090 GPUs cannot train the 4D-NeRF, what is your experimental setting?

[aejion]

We train dynamic NeRF on an A100 GPU. If you need to reduce memory usage to fit smaller GPUs, consider the following advice:

• In 4diffusion.yaml, adjust the settings to: n_levels=8, base_resolution=8, log2_hashmap_size=16, per_level_scale=2, and num_samples_per_ray=128 to decrease memory usage for rendering.
• Additionally, you can reduce the rendering resolution by modifying multi_view.width and multi_view.height.

[Ysz2022]

Are you using an 80G A100? How much memory is required? Is there 70GB or more?

[aejion]

Yes, we use 80G A100 for 4D generation. In our default configuration, it requires about 60GB.

[Ysz2022]

Thanks for your patient response, but now I met such an error when Iattempt to train 4D-NeRF:

Epoch 0: : 0it [00:00, ?it/s]Traceback (most recent call last):
File "/4Diffusion-master/launch.py", line 178, in
main()
File "/4Diffusion-master/launch.py", line 162, in main
trainer.fit(system, datamodule=dm, ckpt_path=cfg.resume)
File "/opt/miniconda/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 520, in fit
call._call_and_handle_interrupt(
File "/opt/miniconda/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 44, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/opt/miniconda/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 559, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/opt/miniconda/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 935, in _run
results = self._run_stage()
File "/opt/miniconda/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 978, in _run_stage
self.fit_loop.run()
File "/opt/miniconda/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 201, in run
self.advance()
File "/opt/miniconda/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 354, in advance
self.epoch_loop.run(self._data_fetcher)
File "/opt/miniconda/lib/python3.10/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 133, in run
self.advance(data_fetcher)
File "/opt/miniconda/lib/python3.10/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 218, in advance
batch_output = self.automatic_optimization.run(trainer.optimizers[0], kwargs)
File "/opt/miniconda/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 185, in run
self._optimizer_step(kwargs.get("batch_idx", 0), closure)
File "/opt/miniconda/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 261, in _optimizer_step
call._call_lightning_module_hook(
File "/opt/miniconda/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 142, in _call_lightning_module_hook
output = fn(*args, **kwargs)
File "/opt/miniconda/lib/python3.10/site-packages/pytorch_lightning/core/module.py", line 1265, in optimizer_step
optimizer.step(closure=optimizer_closure)
File "/opt/miniconda/lib/python3.10/site-packages/pytorch_lightning/core/optimizer.py", line 158, in step
step_output = self._strategy.optimizer_step(self._optimizer, closure, **kwargs)
File "/opt/miniconda/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py", line 224, in optimizer_step
return self.precision_plugin.optimizer_step(optimizer, model=model, closure=closure, **kwargs)
File "/opt/miniconda/lib/python3.10/site-packages/pytorch_lightning/plugins/precision/amp.py", line 70, in optimizer_step
closure_result = closure()
File "/opt/miniconda/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 140, in call
self._result = self.closure(*args, **kwargs)
File "/opt/miniconda/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 126, in closure
step_output = self._step_fn()
File "/opt/miniconda/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 308, in _training_step
training_step_output = call._call_strategy_hook(trainer, "training_step", *kwargs.values())
File "/opt/miniconda/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 288, in _call_strategy_hook
output = fn(*args, **kwargs)
File "/opt/miniconda/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py", line 366, in training_step
return self.model.training_step(*args, **kwargs)
File "/4Diffusion-master/threestudio/systems/fourdiffusion.py", line 95, in training_step
out = self(batch)
File "/opt/miniconda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in call_impl
return forward_call(*args, **kwargs)
File "/4Diffusion-master/threestudio/systems/fourdiffusion.py", line 71, in forward
render_out = self.renderer(**batch)
File "/opt/miniconda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in call_impl
return forward_call(*args, **kwargs)
File "/shuzhouyang/4Diffusion-master/threestudio/models/renderers/nerf_volume_renderer.py", line 82, in forward
ray_indices, t_starts, t_ends = self.estimator.sampling(
File "/opt/miniconda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/opt/miniconda/lib/python3.10/site-packages/nerfacc-0.5.3-py3.10-linux-x86_64.egg/nerfacc/estimators/occ_grid.py", line 164, in sampling
intervals, samples, _ = traverse_grids(
File "/opt/miniconda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/opt/miniconda/lib/python3.10/site-packages/nerfacc-0.5.3-py3.10-linux-x86_64.egg/nerfacc/grid.py", line 161, in traverse_grids
torch.cat([t_mins, t_maxs], dim=-1), dim=-1
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

But I have successfully install all dependencies before, do you know how I can resolve this problem?

[aejion]

This issue might be due to an incorrect version of Torch. Please try uninstalling Torch and reinstalling it according to your CUDA version.

[Ysz2022]

It seems that 4DM can only generate videos with 8 frames right? How can I generate longer videos?

[aejion]

Yes, you are correct. Due to GPU memory usage limitations, we can only generate videos with 8 frames.

[Ysz2022]

Do you have the plan for releasing the training code? Could you please sent the training code to my email?