Theoretically, if you were in a position where a senior member of the team had the authority to make bad decisions, which affected the business in a bad way, and kept implementing these decisions against the will of the rest of the team, and almost behind their backs, what would you do to deal with it?
Be honest and explain the situation to him but be diplomatic. To be honest you are going to have to confront the issue one way or another so you might as well be honest from the start. Try to be non aggressive and convey that you have the companies best interest at heart. Most of all be honest.
well if everyone is being a responsible person (and assuming he is not maliciously doing this) then talk to him. Communicating clearly is important. I think people tend to find subversive ways to deal with problems to avoid confrontation, but that would just cause more problems. Talk to him about the issues.
If they continue happening then you will have to find another solutions, but I would suggest taking the responsible path first... even if it is the harder one and more stressful.
- Talk to him about the problem.
- Do your best to help your boss fix the problems he's creating. In the end, his opinion of you will dictate your work success.
- Document in writing (email) any request to touch production.
- Hope the company realizes he's an idiot and removes him sooner or later. In the meantime, you're protected yourself.
I would recommend that you have a 1 on 1 private conversation with the person.
If that doesn't solve the problem, start documenting the issue in case you need to turn the heat up.
Playing the blame game is not very productive, senior management will respect you much more if you try and deal with the situation instead of just taking steps to make sure you are not blamed.
Talk to him about it.
I had someone do this to my group once. He decided it was a brilliant idea to make a network change in the middle of a trading day dismissing all procedure we had wrapped around such changes so as not to interrupt our users. We were down for about 5 hours because after he realized things were screwed he didn't know how to roll back. I just refused to let it go in front of all the other developers until he was screaming at me. In the end he realized how out of line he was and didn't do it again. It was very confrontational reductio ad absurdum demonstration that worked. I happened to know he was a really bright guy who would eventually come around though. YMMV
Your boss releases directly to production because he is trying to solve problems as quickly as possible. You might be able to streamline your build process to the point that it is just as fast to use source control and do a proper release than it is to copy and paste to the production machine. My team has CruiseControl set up so that checking in a file automatically builds our project, and then it takes 2 clicks to copy that to the production machine and do a full release.
If that doesn't work, you could run a file monitor on the production machine that emails all developers with a diff whenever files change, and makes a backup. That may you at least have a head start on putting out the fire.
I'm of a different opinion. It's not your job to fix your bosses (lousy) performance, in fact it could easily backfire in your face. Even if you are very diplomatic, he could be the type of person that doesn't respond to any sort of criticism from subordinates.
I would try to instead get team buy in for more stringent change-request-review process.
In my opinion the problem isn't so much that your boss is incompetent, but there isn't enough process in place to protect for his ineptitude.
Looking at the problem statement - it sounds like there might be some underlying tension. Could I suggest if this is the case that you consider reorientating the situation. Treat like you would your best friend.
Consider what you'd say to your best friend if you thought it was conceivable you were working with incomplete information and consider it from the view point of what you'd do if you thought your best friend had done something embarrassing, but hadn't realised it.
Ultimately, you need to decide if you're going to say something to your boss. But, be careful about how you frame it. Think of it as giving your boss feedback :)
The problem you have is that you don't have formal controls in place to handle this situation (not blaming you, just stating a fact). First think I would do is to set up a process for changing things in production. I would then remove anybody's access who doesn't need it (like your suggestion everyone but the sysadmins and the senior person's. Actually if you could remove the senior person's access that would be great also, but it sounds like you aren't going to win that battle).
Once you get that in place, I would start keeping a log of all the problems you have and why they occurred (A root cause analysis document) and make sure they get published for people to see. You don't need to blame people in the write up, just explain the issue when it happened, how it was fixed, and what will change in the future to make sure it doesn't happen again. This will start to quickly show why there are problems and who is causing them without overtly blaming them. You can't fix a problem unless you can concretely identify what it is, and although you know how is causing it, I would guess that senior managment is kind of turning a blind eye as to what the real problem is. Once you have the metrics to back up your position you can then start to work on getting the company on board on how to "fix" the issue. A process like this will also help you in the future as if there are any other yahoos who decide to try coding on the fly in production, you have something in place to stop them.
You won't be able to stop your boss from doing is as he has more authority, you can make it uncomfortable for him to keep doing things the way he's done them in the past. This approach will let him come to the realization that it's in his best interests change. He feels good that he's changed for the better, and you get what you want without making him angry.
Your manager clearly doesn't have the interests of the business or your interests as primary concerns. You need to learn to live with that sort of thing, and try to make sure your careers don't suffer. This looks to me like a managerial ego sort of thing.
Talking to him hasn't worked, and he's willing to go around you to install software you don't need. You're not in a good position.
You could try to document what's going on, and it's even possible that it'll make some difference. However, given the situation, I'd bet that upper management won't be interested. Your only real hope is that he screws up enough to be noticed.
I do like the idea of asking for your root access to be revoked. If it's apparent who is messing things up, somebody might just notice. If you can get into a position where you can't clean up his mess, that increases the probability that somebody will notice.
Besides that, keep an eye out for a better job. I've never had a manager that bad myself, so I suspect they're not that common.