I got a RuntimeError, need help plz #29
Description
My setting is torch: 1.10 cuda: 11.3
error:
Traceback (most recent call last): | 0/1856 [00:00<?, ?it/s]
File "train.py", line 211, in
main()
File "train.py", line 165, in main
train_model(
File "/home/sfy/PythonProject/VOTR/tools/train_utils/train_utils.py", line 91, in train_model
accumulated_iter = train_one_epoch(
File "/home/sfy/PythonProject/VOTR/tools/train_utils/train_utils.py", line 46, in train_one_epoch
loss.backward()
File "/home/sfy/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/_tensor.py", line 307, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "/home/sfy/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/autograd/init.py", line 154, in backward
Variable._execution_engine.run_backward(
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [7944, 64]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
When add “torch.autograd.set_detect_anomaly(True)”, the error:
Traceback (most recent call last):
File "train.py", line 211, in
main()
File "train.py", line 165, in main
train_model(
File "/home/sfy/PythonProject/VOTR/tools/train_utils/train_utils.py", line 91, in train_model
accumulated_iter = train_one_epoch(
File "/home/sfy/PythonProject/VOTR/tools/train_utils/train_utils.py", line 46, in train_one_epoch
loss.backward()
File "/home/sfy/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/_tensor.py", line 307, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "/home/sfy/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/autograd/init.py", line 154, in backward
Variable._execution_engine.run_backward(
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [5297, 64]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
How did you solve this problem?
Thank you！