fix broken intersphinx links

[grst]

I don't think it should be necessary to nitpick ignore pandas or numpy stuff. There's probably something wrong with the intersphinx mapping.

Originally posted by @grst in #436 (comment)