tags:

views:

883

answers:

15

A recent chat with some ex-colleagues (who had gone on to bigger and better things) went something like this:

Coll 1 (an admin): "If my app goes down, I'll have to rebuild the databases"

Coll 2 (IT manager in a cosmetic company): "If my app goes down we have to scrap the production run and restart"

Coll 3 (IT manager for high street banking group): "If my app goes down we will be on the evening news!"

Made me kinda glad to be working on mickey-mouse apps!

How dangerous/visible is your job?

+2  A: 

I think you need some more granularity in your rankings but I guess I'm somewhere between 1 and 2.

If I screw up big time, we'd have to restore from backup and deal with the uproar of a couple thousand users.

Michael Haren
+13  A: 

Depends on your definition of "dangerous." In several applications, if my app goes wrong people don't go home. For example in one location it controls the mixing of high explosives. That entire building is designed to direct an interior explosion toward a hill behind it and when running you can't be within a quarter mile of the facility. In another it monitors the loading of large munitions.

I've also done work where it's less severe and a person may only lose a limb (factory machinery can do that).

So if my app fails is a really bad way, it doesn't affect a whole lot of people, but I'd wager that those it does affect are far worse off than most people in your "dangerous" jobs.

ctacke
Wow sounds cool where do you work?
Cervo
We develop and customize products for use in military and industrial manufacturing.
ctacke
A: 

Somewhere between 1 and 2. A few thousand external users will complain loudly, and then while I work on a repair for a few hours, have business managers constantly nagging as to when the application will be back up.

Elie
+6  A: 

I once worked on a payroll system, This went down one friday morning, due to an error in our team.

The Human Resources Manager managed to motivate us to find a quick solution by telling us he would send the workers (welders on oil rafinary, only 500) to our office if he was unable to pay them that afternoon. Payment was in cash, and scheduled that afternoon!!!

For my personal health this was way beyond level 3.

I'm still alive as a prove we succeeded in fixing this.

GvS
A: 

SCM administrator here, for both development data and deliveries (packaged set of files put on the production server).

Since I only have centralized VCS (and not DVCS like Git or Mercurial or...), I do not want any downtime on the server hosting my central repository ;)

That would primarily affect the development, but also any homologation/pre-production test lifecycle based on the versionned deliveries.

The notion of data replication (SRDF), backup and DRP are quite familiar to me in this kind of job.

VonC
+1  A: 

For me, probably between a 2 or 3. I work for a defense contractor, if my app went down for some reason it would upset a few countries, but probably wouldn't make the news.

Ryan Thames
A: 

I've worked on producing safety data sheets for transporting chemicals. If anything went wrong with my software and there was an accident it would have serious consequences.

John Nolan
A: 

If my app goes done, I have a hardware failure. I have to get a new server and install the backup on it. And probably have a phone in one hand with pissed off people and a keyboard in the other hand to fix it.

Paco
+7  A: 

For some examples, you can take a look at History's Worst Software Bugs.

mouviciel
+1  A: 

I write application code for industrial robots, very fast robots which can kill you. While the safety regulations try to prevent anyone from being around them while they run, people can and do ignore them.

Jim C
+4  A: 

I once were on a team developing an application for planning the operation of several hydro-electric power stations. They were totally automated and we sent them orders to open or close the water flow through the turbines. If something went terribly wrong we could potentially drown a village or two.

Jonas Elfström
+3  A: 

I worked at a data center as an network operations analyst. If the servers we housed went down, then hospital networks went down across the nation. I'd like to think I was saving lives by doing my job diligently.

Michael G
+1  A: 

I've worked on autopilots for large container/cruise ships. Our definition of "bug" tended to be far more drastic than the average definition of "bug". They usually involved some set of circumstances involving a combination of human and technical problems that culminate in some Titanic scenario. It helped having a former pilot (ie a person) around to know when things could go wrong. We were uber paranoid and really tried to cover our bases.

However, the key thing in the overall system design was that there was always a manual override. The ship's pilot can always lower one ring of complexity. They can use our system to follow a tracking system from point A to point B (most complex) just control heading (second most complex) to control the rudder angle (next level less complex), or they can completely eliminate our system from the loop entirely and do a kind of "non-powered" steering.

Its kind of a reflection of the whole Law of Leaky Abstractions thingy, eventually you'll just need to go down to the engine room and manually tweak the rudder and its up to the smart pilot to use his/her tools wisely.

Doug T.
A: 

Our software filters a million users' email. So if we screw it up, lots of messages go to the wrong place. Either spam gets through, or clean messages get blocked - either way we're in big trouble and manually recovering would be very time-consuming.

Dangerous? No. Nobody will die, nothing explodes etc. Potentially embarrassing for the company, annoying for our users? Yes.

MarkR
That sounds like a balancing act!
Laura
+3  A: 

My job is not dangerous for me but people living near the zoo where my lion cage gate release software is installed are happy that I write mostly reliable code. The only two escapes were hardware related and nothing to do with the timing algorithms.