How to reduce inference gpu memory?

[yiboz2001]

Hello! I tried using the sota-16z model to reconstruct a 1024x1024x24 frame video with chunk=8, but it requires 94GB of GPU memory. Does the model support multi-GPU inference or tiled-vae inference to reduce memory usage?

Additionally, I tried using chunk=4 for reconstruction, but the results have some artifacts.

[yiboz2001]

chunk=4:

chunk=8:

[yiboz2001]

Hi! Just wanted to check in on the status of the issue I raised earlier.

[henbucuoshanghai]

chunk=8, 4？means ？

[yiboz2001]

chunk=8, 4？means ？

I followed the official guide as below, and the current implementation seems to slice the input video along the time dimension according to the chunk value (for example, for a 24-frame video, it will process it in chunks of 8 frames at a time):

start = 0

while start < num_frames:
    end = min(start + chunk_size, num_frames)
    chunk = video_data[:, :, start:end, :, :]

    with torch.no_grad():
        chunk = chunk.to(device)
        print(f"chunk: {chunk.shape}")
        if text_embeddings is not None and text_attn_mask is not None:
            recon_chunk, _ = model.forward(
                chunk,
                text_embeddings=text_embeddings,
                text_attn_mask=text_attn_mask,
                sample_posterior=False,
            )
        else:
            recon_chunk, _ = model.forward(chunk, sample_posterior=False)
        recon_chunk = recon_chunk.cpu().float()
    output_chunks.append(recon_chunk)
    start += chunk_size

[henbucuoshanghai]

kind and useful answer