Should developers be given Domain Admin / DBA access rights?

Question

I have just walked out of a meeting room steaming after having to unsuccessfully argue the case for retaining our (development team's) Domain Admin / Oracle DBA access privileges. We support day to day production issues as well as a nationally critical process that happens three times a week. On top of this our DBA has no SQL experience and runs most of the production issues they receive through Oracle MetaLink for a solution.

We recently had two DB server downs which were cured by the DBA rebooting the server without trying to locate the issue. When I presented the DBa with a working solution having tailed the listener logs and filtered for offending applications (which we develop) I was vilified for accessing the production system.

What I need is more ammo to prove my case, or (through the beautiful impartiality of Stack Overflow) evidence to convince me otherwise.

The solution put forward by our DBA is to give us a DBA/Domain admin account that we can use at any time to support production. But if this is the case, there is no auditing as it is a shared account and also our dev profiles which you will appreciate are vastly different from a shared one, cannot be used to debug production..

HELP!

Answer 1

It's never a good idea for developers to have full access to production, you should be arguing the case for having a test environment which mirrors the production setup.

Answer 2

I would say that it is never a good idea for ALL developers to have full access, but I think it is important for SOME developers to have limited access to PROD DBs.

For example, if there is a need for simple data correction due to a bug in the program. Sometimes it is necessary to update the data while the bug is being fixed and deployed.

However, usually this limited access prohibits the ability to update key tables like code tables (lookup tables).

Answer 3

Should developers be given Domain Admin / DBA access rights?

It depends on the situation.

On top of this our DBA has no SQL experience

If you have no possiblity of a replacing, and if you trust the developers, then let them have full permissions. Well, same time I wonder why he is a DBA if no idea about SQL. GUI can do the job but still we will have situations only SQL commands can help.

Usually only DBAs have full permissions on Production server (for big firms I will say). And temperory access will be granted at the time of deployment, issues etc. to developers.

Answer 4

As far as everyone has his own login and password, I am for it. And yes, I am a developer. If it is not possible to setup/use separate accounts, I would vote for having one (two) DBA/OPs in charge.

Answer 5

You need a decent change control process in place - one that everyone can agree upon, including the current no-account DBA. People want to control their domains, their little fiefdoms, so be it. But when you become accountable for fixing something "enterprisey", you need a process to follow so that you're still doing your job, and getting the cooperation when you need it, through proper escalations.

But no, I do not think you should have Domain Access/DBA privileges to the database. I certainly wouldn't as a developer.

Answer 6

In a well run organization, there is no compelling reason why developers should have elevated privileges except in development/staging domains and servers specific to their project. Yours, however, does not sound like a well run organization.

If the DBA is not up to the job, the problem should be escalated to someone with authority to rectify the situation. Either he/she must be educated or replaced, or some arrangement must be made so that you can assist him/her properly.

And think about it: you really don't want the responsibility for production environments in addition to your development. You have better things to do.

Answer 7

It sounds like your DBA isn't as skilled as you need him/her to be, and you're trying to use developer access to the production database to get around that.

It's the wrong solution to the wrong problem. The entire development team should not need DBA access to the production database. Fix the real problem; get a DBA you and the development team respect.

Answer 8

A DBA with no SQL experience? I think this is your bottleneck... I have never seen a DBA without SQL knowledge... But probably you will not be able to replace your DBA, so that's no solution...

It is never good to have a lot of people with a lot of access rights on production environment. Better would be to have a development or (or even better: and) testing environment which resembles the production environment.

Answer 9

I agree wholeheartedly with Patrick McDonald. However, somethings which should be noted.

1. There should be a very easy way for developers to access production information. System logs, porgram logs, SQL logs, environment variables and set-up etc. If the developers don't have access to this information, or access to the person who does, then their hands are tied, and you CANNOT expect them to fix anything. (This is something that comes from experience attempted to remotely debug applications I did not write.

2. If the DBA has no SQL experience, that is troubling, but that doesn't necessarily mean that the Developers should be the ones who have access. It means there should be a SQL expert who has access (and is on call for these situations). But that person should be selected specifically to do that, rather than all or any of the developers defaulting to that position.

3. The more people who have access, the harder it is to trace what happens in a time of crisis. If the application goes down, and no one has access, that's a catastrophe. If the application goes down, and everyone starts changing things, that is just as bad. There needs to be a strong chain of command, and also a very strong system of tracing (who made X change) if you DO give some developers access.

4. Giving access to the production environment for developers should never include permission to change the production environment or DBs. The more people who have access, the more likely it is that hotfixes and 'small deployments' will be made without proper regard for deployment procedures, which is how small problems suddenly become big problems. Everything should have to go through the whole deployment process.

Those negatives aside, I believe that there should be a developer or developers with access. But that it should be someone who is given that role. As a developer on a project you should NOT have default access to production, but rather there should be a point-person or someone on call who has at least read access for times of emergency.

Oh, and as a final aside. If it is a shared environment (more than that specific application) then there is a very strong reason NOT to let developers have access. Unless they only have access to the DB that they created, then it opens up the possibility for environment changes that could have far-reaching consequences on all the other applications running off of that DB or server.

Answer 10

I have yet to see a problem that required a developer to have actual access to production systems.

Every place I've been, I've lobbied to stop things like this for no other reason than liability issues. You can't trust anyone, especially not those that work for you. You should know that the #1 cause of data loss or purloined data are employees, not some nebulous hacker.

You need a decent staging environment that mirrors production. This environment must have a similiar amount of data (NOT the same data) and be available for various testing procedures. Further you need to have someone responsible for production.

It sounds like you have that. ALL problems that are not hardware related can be duplicated in a staging environment. These boxes should be identical right down to the patching levels.

Just a month ago, one of our dev leads was insisting on installing the bits necessary to do remote debugging on a production server. Never mind the security implications, he was convinced it was the only way to figure out the problem. While he was arguing with our production team over this, I had them pull the event logs while I ran the test case through our staging server. At the end of the day, the problem was easily duplicated and solved without access to the production servers.

If the production DBA is a moron, then this will show and management will take the appropriate steps to replace them.

Answer 11

Developers should not have production access to change things, they should have select access to diagnose data problems. Any changes to prod data should be scripted and then dba should be fully able to review and run the script. If you dba is not up to the task, then document the problems with this person and get him or her replaced. I submit that if you have no prod rights the dba's incompetence will become apparent fairly quickly.

There are legal and certification related reasons why companies do not allow access. As few people as humanly possible should have prod rights. It is difficult when these rights arefirst removed becasue you have to use more formal processes to promote fixes to prod. Onthe other hand this tend to improve the quality of your changes when you have to formaize them in a script passed to someone else to run. And there should be a reduction of the type of errors that occur when people are making changes on the fly (oops forgot to highlight the where clause and deleted the whole table). And hot fixes that are only on prod but get replaced when dev is next promoted because nobody thought to add it to dev is also a problem that will go away when rights are removed.

I know it's hard to adjust to losing your production rights, but from an organizational standpoint it really is the best choice.

Answer 12

The real problem here is the specific people (the description of the DBA's skills, decisions and self-awareness as shown in the question) and the trust between them and really has nothing to do with the general question of "should (given competent developers and DBAs and Sys-admins) developers administer production?"

Once you internalize that point, then you can focus your arguments to be the most effective. The first question, in face of such managerial incompetence (you fixed a problem and got chewed out for violating political boundaries rather than having it turn the question to the DBA about what he can do to give you access to those logs when needed - that is managerial incompetence) is is this any of your problem? If the DBA is restarting the database server and causing critical things to fail, is the Development group held to blame for that? If not, accept your lack of access and just be very sure that given your lack of access any problems that show up in production are not your problem - argue for a stronger QA budget to make sure problems don't show up in production from the Development group.

If the development group (meaning you) will be held accountable for the production outages, then I would argue for a development representative (one named Developer/DBA) to allow the development group to have a better finger on production problems to allow them to solve them and meet their responsibilities. If possible, don't argue for that person to be you, but someone else you trust.

Also, if the development group will be held accountable even though they are locked out of diagnosing the problem - look for another group or another company to work - this one is setting you up for failure.

Answer 13

In a large corporate environment developers should never have access to any production assets, just development. Here is my reasoning why, and this is based on experience, both good and bad :D

1) If developers have access to production assets and are required to support the systems on these assets then time is taken away from their core job, developing and maintaining code. 2) Developers should be tasked with maintaining code, not servers. 3) As a poster pointed out already, liability. It's hard enough maintaining a large code base, let alone a server. Focus should be on development.

In my world I the only admin access I have is to my laptop. We're busy enough dealing with scope creep and antiquated systems we just don't have the time to do administration. We have very strong change control systems in place too. If a change is made on any production asset without a change control record someone's head will end up on a pike in front of the server room.

If your DBA is without a clue then he and the person who hired/promoted him need to be dealt with. The more your devs support the lame DBA the more you enable him to continue being a failure.

Answer 14

Developers should develop, supporters should support. Mixing them is a bad practice.

If 1st and 2nd line support people can't deal with the issue, they ask 3rd line support (developers) for help. This is an unusual situation and should be treated as an extraordinary event which needs to be analyzed to make sure such problem will not occur again.

Developers usually don't want to share knowledge about the system (especially bugs and limitations). Management tries to lower the risk and get support people to accumulate that knowledge. Supporters like to share knowledge among themselves - that's part of their job.

This is good because it allows developers to do their job without distractions, management to sleep better, supporters to have job, users to have polite supporters to talk to and the system to work better.

Everybody wins.

So decline your responsibility and live in peace.

Answer 15

We give developers two AD accounts - one regular one for daily use and another one with extended rights if they needed to access the production system.

This is a common scenario in a business where the development group that historically has supported the production environment begin to get shut out as DBAs/QAs come on board. In principle, if what the developers do is develop, then they shouldn't need access to the production environment. But often resource is the issue, and there aren't enough well qualified people to look after the production environment and often a developer will need to step in.

It isn't good enough for people to say 'the organisation isn't good enough' because it may well be a small business with a handful of developers and limited resource for looking after production. The ideal scenario is often not achievable or realistic. What should happen though, is for the process to be clearly defined for how an issue is dealt with - including consideration of how to mitigate the risk inherent in allowing growing numbers of people access to the business critical production platform.

Answer 16

Nevah Evah!

Even if developers are well meaning. Here's a story you can even reproduce if you use the Crystal Reports editor:

1. Make a new report based on store procedures.
2. Double click on one of the SP's -> Automagic execution of the SP(just so Crystal Reports can show you the fields).
3. Behold your empty DB.

That last step outcome depends on what the SP does, of course. But I actually saw it happen to an ex-colleague, the DBA had this SP jokingly named "fired", to clean up the database for testing purposes. Permissions? everyone!. Mind you, this was the DB in the testing environment that got cleaned up, but, upon inspection, permissions were the same in the production DB.

So no, the answer is a resounding NO.

Answer 17

Something I learned recently and I would like to share with you: NMFP (Not my problem) :)

Seriously, if your company wants to prevent you (and your team) from accessing some part of your system, why fight it? Just throw everything that comes up on the poor DBA guy. If he can handle it, good for everyone (your team has less to worry about and your bosses are happy), if he can't handle it... Not your problem, you have stated your case, so it's the DBA and your superiors fault, let them handle the problems.

Answer 18

We ran into similar issues here years ago. At first we had one elevated account for developers to access production and then went to individual elevated accounts for the very reason you mentioned (being able to audit changes/logins). In our case the developers are very intimate with all aspects of the system and everyone knows what they can and cannot do if any production issues arise.