Oh, this comes in handy:
vimdiff /path/to/file scp://remotehost//path/to/file
Damn, how many features do I actually know of vim? (thanks, Mika for pointing this out, there's even a blog post about it)
