I have an M1 mac now and I just tried to install Xcode.
The download was fine (I have decent internet at home), but the install itself is mind-bogglingly slow and I was convinced it was just hung.
This stackoverflow answer was useful as a way to sanity check it was making progress and not just stuck. Perhaps we should include a link to it?
https://stackoverflow.com/a/69437989