ClipLoss - RuntimeError: cannot register a hook on a tensor that doesn't require gradient

[Miriam2040]

Hi, thanks for the nice work and great repo!
I changed config to train_with_clip=1 to include ClipLoss.

Then, I am getting the following error in the eval step:

File "/app/CLIP_/clip/model.py", line 347, in encode_image
return self.visual(image.type(self.dtype))
File "/home/miniconda/envs/habitat/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in call_impl
result = self.forward(*input, **kwargs)
File "/app/CLIP/clip/model.py", line 238, in forward
x = self.transformer(x)
File "/home/miniconda/envs/habitat/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in call_impl
result = self.forward(*input, **kwargs)
File "/app/CLIP/clip/model.py", line 209, in forward
return self.resblocks(x)
File "/home/miniconda/envs/habitat/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in call_impl
result = self.forward(*input, **kwargs)
File "/home/miniconda/envs/habitat/lib/python3.7/site-packages/torch/nn/modules/container.py", line 117, in forward
input = module(input)
File "/home/miniconda/envs/habitat/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in call_impl
result = self.forward(*input, **kwargs)
File "/app/CLIP/clip/model.py", line 196, in forward
x = x + self.attention(self.ln_1(x))
File "/app/CLIP/clip/model.py", line 193, in attention
attention_probs_backwards_hook=self.set_attn_grad)[0]
File "/home/miniconda/envs/habitat/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in call_impl
result = self.forward(*input, **kwargs)
File "/app/CLIP/clip/auxilary.py", line 422, in forward
attention_probs_backwards_hook=attention_probs_backwards_hook)
File "/app/CLIP_/clip/auxilary.py", line 250, in multi_head_attention_forward
attn_output_weights.register_hook(attention_probs_backwards_hook)
File "/home/miniconda/envs/habitat/lib/python3.7/site-packages/torch/tensor.py", line 257, in register_hook
raise RuntimeError("cannot register a hook on a tensor that "
RuntimeError: cannot register a hook on a tensor that doesn't require gradient

Am I missing something? Thanks!

[huo852yan]

I meet the the same problem, do you adress this question?

[hucorz]

I meet the the same problem.

[davidfreire]

Hello everyone,

Fantastic work on this repo! Big thanks to @yael-vinker for the amazing repo—I'm truly inspired by your research on sketches.

To address the issue at hand, I suggest modifying the auxiliary.py file located at CLIP_/clip/auxiliary.py. Specifically, you should add a conditional check to ensure that the hook is only registered if attn_output_weights.requires_grad is True.

Currently, lines 247-250 look like this:

# use hooks for the attention weights if necessary
if attention_probs_forward_hook is not None and attention_probs_backwards_hook is not None:
    attention_probs_forward_hook(attn_output_weights)
    attn_output_weights.register_hook(attention_probs_backwards_hook)

Please update it to the following:

# Use hooks for the attention weights if necessary
if attention_probs_forward_hook is not None and attention_probs_backwards_hook is not None:
    attention_probs_forward_hook(attn_output_weights)
    # Only register the backward hook if gradients are required
    if attn_output_weights.requires_grad:
        attn_output_weights.register_hook(attention_probs_backwards_hook)

Also, remember to assign some weight to self.clip_weight; otherwise, it will always be zero.

This should resolve the error.

Best regards,
David