Oh, this comes in handy:

vimdiff /path/to/file scp://remotehost//path/to/file

Damn, how many features do I actually know of vim? (thanks, Mika for pointing this out, there's even a blog post about it)