Skip to content

Stuck at batch = next(data_iter) during training #21

@tututu-7

Description

@tututu-7

When running the train_camera_control.py script, the training process is getting stuck at the line batch = next(data_iter) within the training loop. The code does not progress past this point, and the data loader does not return any batch.
[01/12 16:48:24 AnimateDiff:752]: encoder trainable scale: 199.262 M
[01/12 16:48:24 AnimateDiff:753]: attention processor trainable scale: 18.962 M
[01/12 16:48:24 AnimateDiff:754]: trainable parameter scale: 218.224 M
[01/12 16:48:25 AnimateDiff:768]: Building training datasets
[01/12 16:48:25 AnimateDiff:790]: Building validation datasets
[01/12 16:48:26 AnimateDiff:841]: ***** Running training *****
[01/12 16:48:26 AnimateDiff:842]: Num examples = 321
[01/12 16:48:26 AnimateDiff:843]: Num Epochs = 78
[01/12 16:48:26 AnimateDiff:844]: Instantaneous batch size per device = 1
[01/12 16:48:26 AnimateDiff:845]: Total train batch size (w. parallel, distributed & accumulation) = 1
[01/12 16:48:26 AnimateDiff:846]: Gradient Accumulation steps = 1
[01/12 16:48:26 AnimateDiff:847]: Total optimization steps = 25000

Changes I made: During the video_clips processing, the generated videos were placed in a newly created folder. However, when running the generate_realestate_json.py script to generate the JSON file, it did not execute successfully. So, I moved the MP4 files directly into the video_clips folder.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions