How much time do you spend in production troubleshooting

Question

At my prior job of 5 years, I eventually had to leave because I was growing into the go-to guy for production support and troubleshooting. I was spending at least half of my time doing queries in production, event log scraping, etc. Yes, we had a QA department, a large support organization, and all the right pieces but you're going to get bugs in production, it's just a fact of life for any realistic project team.

Frequently, when the need arose I would be expected to drop everything and troubleshoot a production issue. That's all well and good, in fact it's a primary part of our job as developers who are lucky enough to have a production application in use. However, it's seemingly inevitable that as you prove your abilities as a troubleshooter/debugger, you get more and more of that work funnelled your way.

At my current job, I'm starting to see the same trends: after a production release I am tapped more and more often to triage, troubleshoot, and address a large number of the issues that come up.

So I guess my question is this: when there is a team of developers, and you are getting assigned more and more of the debugging tasks, do you take it as a compliment and learn to deal with the added stress? Or do you start to wonder if you are overburdened with this while your 'real work' for the next release piles up awaiting the next deadline?

(PS this question is not about how to ensure quality of an application -- this is merely typical after-the-fact troubleshooting)

Answer 1

I don't know the specifics about how your company works but one of the better situations I've worked in was when we had 2 teams, one project developers and one production support developers. Developers on the project teams would work on new features and enhancements whilst the production support teams would work entirely on production issues.

The teams were rotated regularly and there was a clear handover between the teams when something was released from the project team into the production support team.

The key was that there was clear separation and it was recognized by management. You can't expect project developers to be productive if they're getting called onto production support issues and you can't expect them to be completely invested in solving the problem when they already having another task to complete.

Answer 2

Quit writing broken code! Duh.

Seriously, though: at some point, you have to learn to delegate. I was on a team for almost five years, from the very beginning of the incarnation of code they were running. By the time I left, I think the "oldest" of my coworkers had been there... maybe 2-2.5 years. Inevitably, that happens: you've been there, you know the code in and out. If something horrible explodes, you're very valuable in fixing it.

If something less horrible happens, though, you have to learn to delegate. Or at the very least, work with someone else to debug it. If you maintain the silo of knowledge, you're going to be stuck being the go to guy forever. If there's enough breakage in your world, it can easily burn you out. If you don't give others opportunity and motivation to pick up some of it, it will never change.

It took a good six months before when I started thinking of leaving, of trying to get other people up to speed on various pieces of the system so there are other people who are experts, if only in one portion of the code, before I felt like the team wouldn't suffer seriously when I left.

Also consider the "what if you got hit by a bus" principle - if you randomly died tomorrow, what would happen to the production environment? Someone else needs to know the system sufficiently well to be able to maintain it efficiently in your absence - that's not negotiable from your perspective, or your company's. And at that point, there's one more person who you can punt bugs to.

Answer 3

I'd take it as a compliment but that doesn't mean its a good thing. :)

If, like it sounds, you don't enjoy it as much as development I would discuses it with your boss and ask him to work with you to distribute the production support workload more evenly among your team. Use the hit by bus pitch that James mentioned if you need to convince him. If you can manage it I would try to establish with him a percentage of time that's acceptable to you to work on production issues such that if it routinely exceeds that number you can bring it up with your boss and discuss why the plan to distribute the production support workload doesn't seem to be working and how to fix it.

On the other hand if you enjoy it or don't mind it particularly just make sure that it doesn't become extra work. I.e. if you had to spend a 5 hours debugging the production sever you can't be expected to meet you usual production for development than week. I would have a discussion with my boss if that didn't seem to be understood.

Answer 4

I used to be in the same situation, becoming known as the go-to guy. Then one day a PM said something to me that made me realize why I was always becoming that guy.

It's ok to say no sometimes.

Sometimes you need to push back on tasks and say, politely, no you can't do that or you can do that so long as this other task they also need is delayed or even handed off to another.

A task tracking system also helps with this. If you have a publicly available system then when people ask you to do something you can add it to the list, or have them add it. And then they can see what your task list is and where their task falls on it.

Also, only accept requests from your immediate manager. If the CEO comes down and asks you to do something, that's fine, but double check with your manager for confirmation. Often they can handball the task on or get the task pushed back up the chain.

Answer 5

The key to stable software(in my humble, short lived experience):

I've found that incremental releases on a fixed cycle works wonders (weekly release), currenlty I have set up an automated crash report system that logs everything to a database (as well as email reports) and a built in self updating system.

Also each increamental release is strictly organised into a very specific set of folders:

Root {Version Number}

New Requests

Bug List

Release Prep

Database Log

Each time the codebase is split off, and the database is backed up prior to each automated release, (this allows for rollback in the worst case senerio)

Some weeks there are no new requests, which leaves plenty of time to fix bugs, many times bugs are found via the automated reports (because users are unaware of exceptions, or sometimes even ignore them). in each release the bugs are the priority, some weeks there are no new bugs, but there may be alot of feature requests. In such cases sometimes requested features are allowed to 'roll' over to the next week. Clients are happy because each release address two things (previous bugs) which leads to a more stable app, and new features which enhance the existing workflow etc.

The above folders information are then entered into a database during downtime, this then allows for client end highlight reports etc..

If you having bugs after bugs, perhaps the design of the entire application should be questioned, bugs to happen but managed correctly over time one would expect an application to tend towards greater stability (not the other way! :)