neural network - Inplace parameter updation without torch.no_grad()

Question

I have just started learning this awesome tool called PyTorch but sadly I am stuck in an equivocal situation.

Below is a code snippet from one of the tutorials

 with torch.no_grad():
       weights -= weights.grad * lr
       bias -= bias.grad * lr
       weights.grad.zero_()
       bias.grad.zero_()

I am kind of confused that even if I will do parameter update without using torch.no_grad() ( i.e. only in-place ) like this:-

 # with torch.no_grad()
   weights -= weights.grad * lr
   bias -= bias.grad * lr
   weights.grad.zero_()
   bias.grad.zero_()

and since the backward call has already been made in the code above this snippet( not included in the snippet ) which basically means all the “grad” attributes are already computed and don’t require the “original” values again. Then, why is it illegal to do those operations without torch.no_grad()?

I know it will flag off the error in PyTorch but I just wanted to know where my line of thought is at fault?

question from:https://stackoverflow.com/questions/65842822/inplace-parameter-updation-without-torch-no-grad

Answer 1

The questions have been solved by one of the PyTorch Moderator on the PyTorch Discussion forum.

Here's the link to it :- Inplace parameter updation without torch.no_grad()