Feedback and Questions

[zazoum-art]

Hello.
I don't know much about technical stuff, I ask o1 to assist me. I tried to install the repo and succeed, but, as I am running on Windows I had to change the dataset_readers.py (the index of directories "/"3 should be "\"3 for windows) and the rearrange.py (there is no cd in windows, the copy should be written for the os another way).
Anyway o1 corrected them for me.
Question:
How much VRAM does the train require? I have a 4090 and cant even do 256x256px 30 frames at 4000 steps. I get OOM all the time. I can't use the examples either, they are too big.

Is it possible to run this on a 4090 win11, and if yes what are the numbers?

[kai422]

Hi @zazoum-art , 4090 is sufficiant to train your input. At which step does OOM occur? Is it during GS training or dynamic prediction?

Also, thank you for your feedback; I’ll make the path operation os-agnostic.

[zazoum-art]

OS: Windows 11 (updated latest)
conda: (latest as suggested
Python: as suggested with cuda 12.4
CUDA: 12.4
GPU: RTX 4090

Fresh PC restart.

Doing clean bear.mp4 from davis videos

====================================================

1. Dataset and flow: SUCCESS
2. Rearrange: SUCCESS
3. train_gui.py: OOM

(das3r) G:\das3r\DAS3R>python train_gui.py -s data/custom/output_rearranged -m data/custom/output_rearranged --iter 4000 --gui
Optimizing data/custom/output_rearranged
Output folder: data/custom/output_rearranged
Reading camera 82/82
Converting point3d.bin to .ply, will happen only the first time you open the scene.
Loading Training Cameras
train_camera_num: 82
Loading Test Cameras
test_camera_num: 0
G:\das3r\DAS3R\utils\pose_utils.py:578: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at C:\cb\pytorch_1000000000000\work\torch\csrc\utils\tensor_new.cpp:281.)
grid = torch.tensor([xy_grid(W, H) for _ in range(len(depth))] , device=depth.device)
G:\das3r\DAS3R\scene\gaussian_model.py:620: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
conf_static = 1 - torch.tensor(dyna)
G:\das3r\DAS3R\scene\gaussian_model.py:622: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
confidence = torch.tensor(confidences).reshape(-1)
Ratio of confidence masks: 0.1552
Ratio of aggreagted masks: 0.1552
Number of points before: 12091392
Number of points at initialisation : 1876009
Training progress: 0%| | 0/4000 [00:00<?, ?it/s]Traceback (most recent call last):
File "G:\das3r\DAS3R\train_gui.py", line 670, in
training(lp.extract(args), op.extract(args), pp.extract(args),
File "G:\das3r\DAS3R\train_gui.py", line 520, in training
iter_report = training_report(
^^^^^^^^^^^^^^^^
File "G:\das3r\DAS3R\train_gui.py", line 596, in training_report
image = torch.clamp(renderFunc(viewpoint, scene.gaussians, *renderArgs, camera_pose=pose)["render"], 0.0, 1.0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "G:\das3r\DAS3R\gaussian_renderer_init_.py", line 126, in render
shs = pc.get_features[filtering]
^^^^^^^^^^^^^^^
File "G:\das3r\DAS3R\scene\gaussian_model.py", line 190, in get_features
return torch.cat((features_dc, features_rest), dim=1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 344.00 MiB. GPU 0 has a total capacity of 23.99 GiB of which 0 bytes is free. Of the allocated memory 22.46 GiB is allocated by PyTorch, and 282.37 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Training progress: 0%| | 0/4000 [00:02<?, ?it/s]

================================================

Tried bear.mp4 clean with converting it to animated .gif ( OOM), png sequence (OOM)

MAYBES: Maybe I messed-up with the script. Maybe its an OS error (saves as .npy on output -what's a .npy?). Maybe the steps are wrong. Many maybes.
Doing REMOTE on Huggingface space: got error. Now it is down.
GRADIO app.py: Same.

[kai422]

Hi, I think there are some issues with the initailization points, 256x256px 30 frames will at most give 196k points, but it shows the total points is before conf thresholding is 1209k.

I will do some testings on windows and let you know.

[zazoum-art]

It is NOT 1,2kk point it's 12kk points. o1 says the points are too many.
12091k points is the bear.mp4 demo. Your clip. I debugged up here with one of your clips.

bear.mp4

[zazoum-art]

OK, FOUND IT

I messed up with the train_gui.py. Now it works.

BUT

You still have to make dataset_readers.py and rearrange.py windows_os-frendly. Do you want me to PULL REQUEST?