Server farm Shutdown

Question

How do large server farms handle gracefully shutting down all or part of the farm? I'm thinking of planed and unplanned cases like:

• "We need to shutdown Rack 42"
• "We need to do work on the power feeds to the whole block"
• "Blackout! UPS's running out of Juice! Aahh!"
• "AC is down, air temp is 125F and climbing"

The issues I'm interested in are how people handle sequencing, and kicking the whole thing off. Also it occurs to me that this could easily get mixed with bringing up and down services and with the software up grade system.

(At this point I'm more asking out of curiosity than anything.)

Answer 1

One method is to mirror the live machines on temporary hot-swaps and, assuming access is via network, cut over by reconfiguring the router to divert traffic to the mirrors. This process can be automated for unplanned outages.

For planned maintenance, some simply notify their users that the system will be unavailable during a certain window.

Redundant power supplies and gas-powered generators handle most power-related problems, again with automated failover.

Answer 2

Computers can use a lot more power coming back online than they do running, since they have to get all of the platters and fans spinning, typically have heavy CPU activity starting all of the applications, and so on. Most shops will have a set sequence that staggers the startups, so they don't max out the circuit and have to start all over again. This is also important if you have a bunch of applications that expect to talk to a database, or a bunch of web servers that need to talk to the app servers. You usually start from the bottom up, and stagger the startups by 30 seconds to a minute, depending on how many boxes are on your circuit.

Answer 3

Ah, now I understand your question more clearly.

Products such as the iBootBar from dataprobe allow you to monitor and manage the power to remote devices. An intelligent system can monitor the current draw of each device to verify that it's functioning within nominal limits. If not, it can take the equipment offline and bring a spare online to replace it, watching for the initial surge and waiting for power to stabilize before switching the next device on.