Consulting and the Joel Test

Question

The Joel Test sounds like a list of attributes I'd like to work with (and doesn't it for most of us?) But in a consulting context, it can vary a lot. I've been told it depends on the customer, which in some cases do not even have source control (egad!)

Is it justified to turn down a consulting job on the basis of potentially low Joel Test score in some situations?

Also, how could a low Joel Test score be remedied? Is on-the-go version control feasible (say you have it on a laptop you bring on every job)? Would that be accepted everywhere? Ideas? Annecdotes?

(Making this a community wiki from the get go, as it is obviously very subjective)

Answer 1

You're free to turn down all the consulting jobs that don't pass the Joel Test.

You're also free to starve.

Pick one.

EDIT

I thought of a good way of illustrating the choice. Don't get me wrong, I think the Joel Test is a great metric for trying to achieve the ideal work place. It is just that Fog Creek is only a very small vertical portion in the very horizontally wide software market.

So here is what I would imagine is a typical support call to Fog Creek. (I've never worked there; I don't mean to imply that they are not extremely hard workers and don't have stress. Clearly, they have some talented people there. This is for illustrating a point, nothing more.)

RING - RING

Support: Fog Creek support, how can I help you?

Developer: Hey-o, our bug tracking system is down for some reason. Can you help?

Support: Sure, try this, this, that, maybe this, then that other thing. Does it work?

Developer: No, but this widget here displays "Stack Overflow in call to write()" when I click it.

Support: Oh, I see. Well I'll have to escalate this to our developers. They should have it fixed in a few days.

Developer: Sigh...ok, I guess we'll just file reports with Earl and he can manually key them in when the system is back up.

Then the Fog Creek developers get together, discuss the approach to fixing the bug in the hallway (item 12), discuss how it fits in with the spec (item 7), then tell the project manager that the new features need to stop so that a bug can be fixed (item 5) and that the schedule must be slipped (item 6). They then put the bug in the bug database (item 4) from the comfort of their sound-proofed offices (item 8). They then fix the bug using the latest and best tools (item 9), check it into source control (item 1), check that the system still builds from typing "make" from the code root (item 2) and the build system picks it up during the next build (item 3). The QA department tests it (item 10) and releases it to the customer ONLY if it passes.

That works great and I only wish I could work somewhere like that one day. But change the business to a software company that makes software responsible for companies collecting money. It could be anything from the e-commerce shopping cart, their portal CMS, payment processing software, etc. Then the rules of the game change. Here is a typical support call in that situation.

RING - RING

Support: Financial software creating kompany (FSCK) support, how can I help you.

Analyst: Hi I am an A/R analyst from Megacorp and OMFG, OUR WEBSITE THAT SELLS OUR WIDGETS CAN'T ACCEPT PAYMENTS!!!! FIRE!!!! FIRE!!! ARRRRGGGHHHH!! HACK *COUGH* SPIT

Support: Er, um, well what is it doing exactly?

Analyst: ARRGGH!!!! I DON'T HAVE TIME FOR THIS JUST MAKE IT WORK!!! I DEMAND THAT YOU MAKE IT WORK NOW!!!!! I AM GOING TO CALL YOUR CEO AND RAT YOU OUT IF YOU DON'T MAKE IT WORK NOW!!!!!

[cue developers scurrying in looking disheveled because they just woke up after sleeping under their desks all night trying to deal with Initech's problem from last night.]

Developer: What seems to be the problem?

Analyst: We made a change in our system last night and nothing seems to be working anymore.

Developer: What did you change? Do you think it had anything to do with what is occuring now?

Analyst: HOW DARE YOU SUGGEST THIS IS OUR FLAW!!!! WE TEST THINGS OVER AND OVER AND HAVE A COMPREHENSIVE CHANGE MANAGEMENT PROCESS!!! IT COULDN'T POSSIBLY BE OUR STUFF BECAUSE WE JUST SPENT 3 MONTHS ROLLING OUT THIS NEW THING TO OUR ENTERPRISEY SYSTEM!!!! I DEMAND IT WORK NOW OR I WILL CALL YOUR CEO!!!

[developers scurry off to look at the code to see if there were any recent changes that might exhibit the behavior described by the less-than-cooperative analyst. it takes too much time, so....]

RING - RING

CEO: Hi this is Jerry, the CEO of FSCK what seems to be the problem?

Analyst: Your developers and support won't help me. My widget selling system has been down for an hour and I haven't even heard about the fix.

CEO: I'll get to the bottom of this.

[cue CEO jogging down the hall to the developers offices]

CEO: WHAT SEEMS TO BE THE PROBLEM? WHY AREN'T YOU TAKING CARE OF OUR CUSTOMERS!!!!! FIX IT NOW!!!

[ceo jogs off sipping a soy latte that magically appeared in his hand when he snapped his fingers]

[developers look at each other, look at the screen, look at each other, then run for their keyboards and frantically start typing in code]

Developer 1: THIS CODE HAS THE POTENTIAL FOR A BUFFER OVERRUN!!!! THIS OTHER PIECE HAS LOGIC THAT WILL NEVER WORK!!!! YOU INCOMPETENT FOOL, SEE WHAT YOU HAVE DONE!!!!

Developer 2: SCREW YOU!!! I HAD TO PUT THAT IN THERE TO HACK OUT THE 16-LEVEL DEEP NESTED LOOP YOU HAD IN THERE THAT CAUSED INITECH'S PROBLEM LAST NIGHT!!!

Developer 3: Too bad they didn't buy that code analysis tool. The CFO couldn't justify a $5k tool that does the same thing as having developers read the code. We would have found those things pre-release if we could have had some analysis done on the code....

Developer 1 & 2: SHUT UP AND START CODING!!!

(so much for Agile's mantra of practicing group ownership of the code eh?)

[developers hack a while, check the code in, get a build (items 1 and 2...because those are the easiest)]

Support and CEO: NO TIME FOR TESTING, THE CUSTOMER IS PISSED!!! TELL THEM WE'LL GIVE THEM THE BUILD AND THEY CAN TEST IT IN PARALLEL WITH OUR QA DEPARTMENT.

[cut to view of QA department playing darts and joking about how they'll get right on that parallel testing for Megacorp's patch]

Analyst: OMFG!!!! THE PATCH DIDN'T DO ANYTHING!!!! WTF BBQ!!!!!! ARRGGGHHHH!!!!! I HAVE AN ULCER NOW FROM OUR ENTERPRISEY SYSTEM GOING DOWN EVERY DAY FOR THE LAST 5 YEARS, AND NOW YOUR SYSTEM IS RESPONSIBLE FOR ALL MY PROBLEMS!!!! AND I EAT TOO MANY SONIC JALAPENO BURGERS FOR LUNCH TOO!!!

[switch to developers running out into the dark parking lot. they keep shoving each other out of the way as they dash for their cars]

Developer 1: Out of my way! I need to get in my car and get the hell out of here before the CEO catches us and makes us stay all night again.

Developer 2: Last one out of the parking lot is the CEO's slave tonight!

[switch to view of CEO sitting in a strip club with a bunch of other CEO look-alikes, smoking a cigar while holding a glass of Jack Daniels]

CEO: HAR HAR! When I left them, they were madly hacking away to fix the problem. They should be finished sometime this week but I told them they weren't allowed to leave until it was fixed! Suckers!

[switch to morning view outside FSCK. all looks normal, birds are chirping, people walking their dogs past FSCK's front lawn. they don't pick up after their dogs do their business on FSCK's front lawn. respect! one of the new developers is sleeping on the front lawn using a copy of Code Complete as a pillow]

[switch to inside view of tech support representative. blood-shot eyes, steam coming out of the ear that isn't covered with the headset's ear cup]

Analyst: Everything seems to be ok now. It turns out that one of our outsourced partners updated a line of code that caused an exception to be thrown every time someone tried to buy a widget. We couldn't see the exception because IT had also made a change to some of the switches between our production server and our syslog aggregator. It turns out the syslog traffic was being routed to the firewall where it was being blocked rather than routed to the syslog aggregator. Thanks for your help. CLICK

[ceo jogs in]

CEO: Did we patch Megacorp last night? Are they working now?

Support: Yes, they are working now.

CEO: Excellent!

[ceo snaps his fingers and another soy latte appears in his hand as he turns around a jogs out. The support representative's head slumps over and hits his keyboard as he passes out in exhaustion. fade to black]

Unfortunately, it is like this in every company I've worked at in the past dozen years or so. I believe this is close to reality in most companies. Had they started out passing Joel's test, I believe it would eventually deteriorate to this when it came to keeping paying customers happy. I've worked in a company that was chaos when I started, it moved to a more ordered development process and probably was within a year of passing most of Joel's test points. It then went back to chaos as someone decided to change the revenue goals. Non-developers usually do not care about process nor do they see value in keeping developers comfortable and productive.

This clearly has to change. I'm hoping one of these top-tier MBA schools decides to offer classes on how to run a profitable software company. Maybe then it will catch on that making software and being profitable requires more than just adding people to the sales and marketing teams. Until that time, I don't expect much will change. Software companies will continue to come and go and everyone will still believe it was mostly based on the failure to pump out code fast enough to beat the competition.

Meanwhile, you need to make money consulting. There are many more FSCKs out there than there are Fog Creeks. Finding the FSCKs are easy; just drive by in the morning and see if there are developers sleeping on the front lawn. Finding the Fog Creeks is not so easy. Holding out for the hard to find companies just doesn't pay the bills.

Answer 2

Most jobs I had that filled 8+ of these tests had no need of consultants.

Clients (from my consultant years) either have no need for the 12 (quick contract) are not interested ("I'm paying you to code, so code") or if your lucky will be happy to listen and help you bring such a system up, and you should have a permanent job offering near the end.

The best thing about being a consultant is to be able to choose who you work with. The #1 reason to refuse another contract with a client is the way he previously treated me, and that includes how I can apply good coding practices. Guess who get's blamed when everything is rushed out with no spec, minimal testing and beta and pirated development software. At first it generate more job (support calls), but the client will soon complaint how things are never done with.

Answer 3

The Joel Test roughly rates the quality of a software team. You can do things as an individual to try to raise a low test score, but that won't change the basic problems endemic to a particular team. If a software team doesn't use source control, you can be damn sure that they are going to be seriously dysfunctional in many other ways.

Many companies that need to hire consultants are not exactly going to score highly on the Joel Test in the first place. That said, as a consultant you might find yourself in a good position to positively influence that team - you could be the person who installs SVN or git somewhere and convinces everyone to use it. Sometimes a bad team just needs someone with new ideas to help improve things.

You have to decide for yourself where you draw the line on the Joel Test. Personally, I would NEVER accept a job at a place without source control unless they were literally dumping truckloads of cash at my front door, and even then I might give it a second thought. It's just not worth the stress.

Answer 4

In 30+ years of consulting, almost none of my clients have scored more than 1 or 2 on the Joel Test. A few scored in the high 8's, but those were the exception, not the rule.

"Is it justified to turn down a consulting job on the basis of potentially low Joel Test score in some situations?"

You can turn down anything you want for whatever reason you want. No one cares about the justification.

Seriously. You're opinion doesn't count for anything.

Clients who are desperate for staff will not care why you turned them down. Your rejection will not lead to sudden moral crisis where they rethink their mistakes.

Your opinion of their development practices doesn't matter at all. Your reason for turning them down doesn't matter. You don't need to "justify" anything.

Indeed, they'll usually laugh if you explain why you turned them down. They know that what they are doing is the level best under their circumstances. They absolutely know that they cannot -- for example -- use source code control because they don't have the time or budget or server space or some other ridiculous excuse.

You can point it out all you want. They generally won't care. They can't care, since they know what they're doing is already ideal under their unique circumstances.

"Also, how could a low Joel Test score be remedied?"

It can't be. A culture that performs poorly, will continue to perform poorly until significant changes to the culture and reward structure are made.

One way to effect change is to work and make the case from inside the organization that things could be better. If you're successful, some folks may seek to emulate what you're doing that's successful. Turning them down does not demonstrate successful software development practices.

"Is on-the-go version control feasible?"

Yes.

I have a laptop I bring on every job.

"Would that be accepted everywhere?"

Mostly. A few locations get nervous about consultants bringing in "outside" devices. They also complain that video and sound recording devices are strictly forbidden, yet iPhones are allowed. So, if they want to create trouble for you, they can.

Some places won't let you build code on your laptop. Some will let you.

Answer 5

This is similar to the questions from direct employees about introducing better process (or better agile) in an environment when you don't have buy in from management.

I think its easier to improve things on ones own without buy in from management if the issue is benign neglect ("Source control, what's that?") and not active sabotage ("I will not pay a cent for any time spent on bug tracking, source control, unit tests or build automation!")

Some process improvements can be done ones own. Run an issue tracker and subversion on your own machine and track your own work. Use portable apps like XAMPP to host apache and any php bug tracker if you need to, or an internet accessible bug tracker and source code host if the client isn't specifically prohibiting it. If they don't pass the Joel test, they're clueless enough to not be able to micromanage you, so you should have the flexibility to automate your build, using TeamCity or Luntbuild if the're isn't any money in the contract for tools. Most clients want developers to be in the loudest environment possible, so invest in good headphones-- some headphones can block up to 20 decibels of background noise.

Even Joel (on one of the SO podcasts) has said that specs as a communication tool promise more that they can deliver. If the client is failing on everything except having a spec, then I wouldn't trust their spec to be useful either. You can code to the spec, but that won't make them happy because it takes a sophisticated customer to know what they want in detail when buying custom software. A contractor can always opt to write a spec, it's a just a matter of time and will you be able to bill for it.

The rest of the Joel test items are management issues that individual (be it a contractor or direct hire) initiative can't influence (outside of a non-binding recommendation)-- budget, interviewing process, office layout, who's available for testing, etc.