I have a system where a website needs to be hosted on a Linux machine while a backend application that the site talks to needs to reside on Windows. Is there any "common practice" for such hosting?
Note - both of the systems are mine so there is the dilemma of whether to have the machines physically located together to avoid delay for calls over the net.
This is a great scenario for web services. You could set up a service (using WCF, JAX-WS, etc.) on the Windows side, and a client in whatever the language the Linux side will be using. That's the beauty of web services: interoperability.
There's a web hosting service that provide both windows & linux. It's called Mosso.
EDIT: From what I can see, I think it would better you use services like Mosso, however, if you can't, you can have both services running at the same time and have it talk to each other via sockets. I suspect latency is going to be a problem if you have a site as busy as stackoverflow. If you do, it's probably better to have both servers located in the same place physically, connected by LAN.
Depending on what language the systems are written in, you could run them on the same server, either a windows or linux server - PHP runs well on windows, and with the use of Mono, you can get most .Net apps running under Linux too
(As an aside, I'm posting as a favor to statenjason above.)
Although you didn't mention the specific numbers for the servers, I'm assuming both are relatively modern machines and capable of the following--
As you describe the Windows application as a "legacy backend", I'm assuming that it was written for, and likely prefers running on an older version of Windows (XP or previous). As these systems are becoming more prone to security holes, it is important to distance them from the open network (or similar attack vectors).
For this reason, as well as to minimize the distance between the RoR app and the backend, I would recommend virtualizing the entire Windows install inside a Linux server. This allows the Linux server to act as a frontend while still using the backend inside Windows through the loopback network device (faster than having to run through an external router). The less powerful of the two servers could then be used to mirror this set-up to provide a backup and failover solution.
Virtualizing Windows also allows easy recovery and protection, while minimizing the need for conflicting applications to be running (i.e. firewall, antivirus).
To answer whether the machines should be located together, in a backup setup as mentioned previously, no, and in a networked computing setup (both hosting individual apps), yes.