One thing that bugs me about some of the datacenter hosts that I have to access for one reason or another under a generic developer account that has no write access to anything except /tmp is that there is a long delay while xauth tries to lock $HOME/.Xauthority. I've explicitly unset DISPLAY, but that makes no difference.