I'm using a single handle for the entire project.
The game object is deactivated and reactivated as required.
With only the Update loop detecting changes, this isn't executed when the object is disabled.
If the object is selected, moved, deselected, selected and then deselected again, the object is being moved to the position of the deselected click position, as it's not picking up the change due to it being deselected and disabled.
I'd rather not completely delete and recreate the handle object each time, as there is only one ever used in the scene and that slight overhead is a little bit of a pain.
Adding creation and Destruction of the handle object during the OnEnable/OnDisable lifecycle fixes that it doesn't get to see the change during the Update function.
Alternatively, have an explicit "SetTransform" function to handle it?