Skip to content

Conversation

@manugaurdl
Copy link

Bug

Data mismatch error in torch/region/fourier_features with input tensor being FP16 and frequency matrix beign BF16

Error trace

File "/home/manugaur/moondream/moondream/finetune/finetune_region.py", line 179, in main
    encode_coordinate(bb[0].unsqueeze(0), model.region),
  File "/home/manugaur/moondream/moondream/torch/region.py", line 43, in encode_coordinate
    return w.coord_encoder(fourier_features(coord, w.coord_features))
  File "/home/manugaur/moondream/moondream/torch/region.py", line 28, in fourier_features
    f = 2 * math.pi * x @ w
  File "/home/manugaur/vlm_obj_det/.venv/lib/python3.10/site-packages/torch/utils/_device.py", line 103, in __torch_function__
    return func(*args, **kwargs)
RuntimeError: expected mat1 and mat2 to have the same dtype, but got: c10::Half != c10::BFloat16```

Patch

encode_coordinate(bb[0].to(model.region.coord_features.dtype).unsqueeze(0), model.region),
encode_coordinate(bb[1].to(model.region.coord_features.dtype).unsqueeze(0), model.region),
encode_size(bb[2:4].to(model.region.coord_features.dtype).unsqueeze(0), model.region).squeeze(0),

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant