We are a small group of 8 to 12 developers. We have clients that run web applications. Some are starting to ask for 24 hour emergency support. What is the best scheme for having employees share on-call duties?
What compensation should be provided?
I would be happy to hear what good schemes you have witnessed and experienced.
Have a rotating call list with an On-Call person and a Backup to the On-Call person. Rotate it every Monday AM. If calls are light, then rotate every 2 weeks. Compensation should be based on how many interruptions occur while on call. Where I used to work, same size as you speak of, the On-Call person also handled emergencies during 9-5 hours too. I like to make compensation in the form of cash. My employees have bills to pay and no amount of thank you's will be able to pay their bills.
At least in support circles, the general policy is to have an after-hours phone/pager that gets rotated among the team. Whoever carries the phone gets paid additional income, which can vary but I've heard anywhere from $250-$800 per week for pager duty. IMO, it's also preferable to let the team manage this among themselves with minimal management oversight, so that the team members can arrange the pager duty to fit personal schedules. After all, as long as someone is carrying the pager at all times, it really shouldn't matter who, since they're being compensated for it anyway. Depending on level of reliability desired, it may be necessary to have a "backup" on-call person as well.
Rotate people in on a weekly basis. With 8-12 people, they're not going to be on call that often. Cycle more quickly and it can become difficult to keep track of who is supposed to be on call.
Offer overtime (1.5x or 2x pay) for events when someone does get called, from the time they receive the notification to the time they are back at home (or wherever) and done with it.
You could also pay a bonus simply for being on-call, and make it an opt-in system, so those developers who want the extra cash could participate, and nobody feels compelled into an on-call position they didn't sign up for. But whether that's workable depends on your group's culture.
We do a weekly rotation between the members of the group. There are only 4 of us on call, so it ends up being once a month. In your situation, with 8-12 devs, it would be even less often.
The way ours works is, the on-call person gets the shared cell phone (though we normally just forward it to our personal phones, so we don't have to carry two). Then, that person gets the first call when something needs immediate attention. They can either solve the problem or get in touch with someone who can.
As far as compensation, we don't get anything extra, but I've heard of companies doing several things, such as extra pay during the on-call period, extra time off to compensate for being on call, etc.
Oh also, we have weekly meetings where we hand off the phone and go over any issues encountered the last week, so people know what kinds of things were handled. This prepares them for their turn, and also gives us a chance to recommend any changes that may prevent further issues in the future.
Consider what effects being on-call has apart from just carrying a pager.
Do they have to be able to drive to a site - so you can't drink while on-call.
Does it involve them having to be in front of a computer within a certain time - so can't be off hiking/sailing.
This effects if there should be some form of payment for simply being on call, even if there is no call.
You need to pay people for being on call. My Rules:
Now, the compensation should be geared to the likelihood of there actually being an incident.
If, for the past 100 days, you've had one event, then I'll take a minor bonus for the week ($100 ) and an incident premium if something occurs ($500 for getting out of bed.)
If however, you've had incidents 50 times in the last hundred days, then I'm going to want some serious compensation for taking the pager, but I'll waive the incident fee.
However, If I'm coming into the office at 4 am to press the magic button, then I'm getting a day off (preferably the next day).
If somebody is up half the night due to a server blowing up or something, getting up early to get into work for 9am or whatever won't be too appealing. It could also be potentially dangerous if they're driving having had little sleep, and their productivity will be lower. I would therefore recommend you allow your developers who are disturbed in the night the option of taking time off in lieu the following day ie work 4 hours overnight, come in at lunch time the next day.
Of course, as working out of hours is overtime you should probably pay them for the extra hours worked as well as the time in lieu, although the time in lieu means you probably only need to pay them their standard rate rather than 1.5x/2x.
Why not consult with your guys and see what they prefer? After all, as they're the ones involved it probably makes sense to take their views over those of others, as long as you're happy with what they propose.
Compensation depends on the likelihood of getting support calls during the time someone is on call, and can be provided as either monetary, time-off-in-lieu, or a mix of both.
If there are two people on call every day of the week (non-work hours), one person from 1am until 9am (then work hours) then 5pm until 1am, then the person on call is effectively "at work" for another shift each time they do it - their freedoms have been severely curtailed.
The level of curtailing is another thing altogether. Can they sort the problem out at home via computer? Can they provide support over the phone without a computer at all?
If over a year someone is on call 50 times, and they work 225 days a year normally, then they need to have compensation equivalent to 2/9ths of their wage multiplied by an "on-call" factor between 0 and 1 that is determined by the expected level of support calls for the person on call, and the shift hours they are covering.
If that factor ever gets above 0.3 I would consider hiring full-time support staff, as their wage is lower than the developers, and your developer time is better spent developing.
What about putting everyone on call, all of the time? This works very well at a friend's firm (financial software). This avoids the problem where a developer who knows how to fix the problem (or created it) is off the rota the week that the problem occurs.
You might want to consider whether you should be compensating those who get called in to support on these issues. If these are the developers creating the bugs, then the problems will soon go away when they have to fix them in the early hours.
The biggest problem with implementing the strategy above occurs if you need the employees to answer each and every support ticket outside of normal hours. I would suggest that you employ someone else (or another organisation) to handle the initial contact with customers and then escalate to your normal employees. This person would just operate outside normal hours and deal with support issues, or other work you have in-house, perhaps with the option of moving onto the development team.
You also want to be supportive of the employees by implementing systems (bug tracking, source control etc) that help make this support process easier. This is important if you follow the everyone on call strategy, or any other.
Speaking strictly for myself, if I were told to be on call, I'd start looking for another job. I didn't apply for a support or admin position, I applied to be a programmer. Hiring someone to do one job, then telling them later that they'll have to do another job, is nothing more than a bait-and-switch.
This service is good since it allocates support staff automatically with minimal intervention: www.onyaka.com
Where I worked, we did the following
Hourly people got 14 hours a week for being on call, and a minimum of 4 hours per call-in, unless the call overlapped the normal working hours.
salary people got 200 bucks each on week on call, and 100$ per call in.