Can we improve the Joel Test?

Question

I was recently answering a question referring to the Joel Test and found that our company did not do well on some points (hallway usability testing and daily builds) but that these points for our company are relatively minor, while I agree that some points are an absolutely must (source control, bug database, interviewees writing code). Now, the Joel Test is good due to its simplicity and I am not really asking to replace it. Nevertheless, are there some points that you find are missing in the original test or some that should not be there? What weight would you give each point?

---

For reference, here is the original test:

1. Do you use source control?
2. Can you make a build in one step?
3. Do you make daily builds?
4. Do you have a bug database?
5. Do you fix bugs before writing new code?
6. Do you have an up-to-date schedule?
7. Do you have a spec?
8. Do programmers have quiet working conditions?
9. Do you use the best tools money can buy?
10. Do you have testers?
11. Do new candidates write code during their interview?
12. Do you do hallway usability testing?

(I will write my own answer into the answer section, since you may like the question but not my answer.)

Answer 1

I find that the points may need to be weighted:

1) Do you use source control? [+++]
2) Can you make a build in one step? [+]
3) Do you make daily builds? [+]
4) Do you have a bug database? [+++]
5) Do you fix bugs before writing new code? [++]
6) Do you have an up-to-date schedule? [++]
7) Do you have a spec? [++]
8) Do programmers have quiet working conditions? [+++]
9) Do you use the best tools money can buy? [++]
10) Do you have testers? [+++]
11) Do new candidates write code during their interview? [+++]
12) Do you do hallway usability testing? [+]

Here are some additional points for starters, also weighted:

a) Do you have a dual screen setup? [+]
b) Do you have to work overtime a lot? [---]
c) Do you have the option to work from home occasionally? [+]
d) Do you learn new technologies in your work? [++]
e) Does the company have a high turnover? [---]

Answer 2

some additional ones with explanations:

• Do you do peer code reviews as a point of learning & sharing? Example: Once a month, developers sit in a room (2 hours, during a company paid lunch) and discuss something about their code:
  • Some code they refactored
  • Some code they optimized
  • Architecture improvements or new design approach
• Is there time for learning? Not in terms of learning the code base but allowing for developers to spend few days actively learning about a new technology or design concept. Such "non-productivity time" would not be looked down upon by management or non-developers.

Answer 3

Similarly to the OP's weighting proposal, I'd suggest more of a staged progression approach, reordering the questions:

 1. Do you use source control?
 2. Do you have a bug database?
 3. Do you have an up-to-date schedule?
 4. Do you have testers?

No company has any business writing software without this lot. If any of these fail, you're done with the interview.

 5. Do you have a spec?
 6. Do you use the best tools money can buy?
 7. Do you fix bugs before writing new code?
 8. Do you do hallway usability testing?

If these fail there should be a good reason for it, but there could be one. Too many fails would be a dealbreaker.

 9. Do new candidates write code during their interview?
 10. Can you make a build in one step?
 11. Do you make daily builds?
 12. Do programmers have quiet working conditions?

And any of these failing would be a worry, but probably only damaging depending on the domain the company is in - quiet working conditions could be a negative for a creative focused web-shop, daily builds is going to be very product dependant.

I also reckon the list needs to ask serious questions about project methodology, TDD, how/who/where support is conducted, and their view on dedicated research time.

Answer 4

There were a few points raised in this thread.

Answer 5

I second the point about peer reviews. We actually go a stage further. Peer review is carried out everyday, and all code is reviewed. Every coder reviews somebody elses code. OK, initially, this took an hour out of everybody's day, but it's a lot less than that now. We also have far fewer issues in internal QA and UAT. We also deliver on time much more often now.

There are some subtle advantages to doing it this way:

• By having all code 100% reviewed, we write better code. Full stop, no question.
• As a senior developer reviews a junior's code, the senior can get a real handle on the capabilities of the junior, and the junior gets to learn from his/her mistakes
• As junior developers get to review the code of senior developers, they get to learn best practice, style etc.

One point that Joel makes that I don't agree with (can you say that????) is the one about developers having their own office. I've found that OK, some people get into the groove some of the time, but all people loose focus at some point each day and just drift, surf the web, email their mates etc.

We use an open plan office, but all developers have ear phones so they can get "quiet time", and we have a rule that you never ask a question that you can answer yourself via google or documentation.

Answer 6

Based on Joel's article about the Joel test, I would assume two things about it:

1. It was meant to be simple and easy.
2. It's not necessarily meant to judge if a place is a good place to work. It just judges if the organization is getting its development process right.

So it's not a perfect test and wasn't intended to be: "The bummer about The Joel Test is that you really shouldn't use it to make sure that your nuclear power plant software is safe."

With that in mind, I would argue that if anything needs to be done to improve the Joel test, things should be removed rather than added. There are an infinite amount of things that could be added to make it a better test. The problem is that the more you add, the more you defeat point 1.

However, I can't think of anything that can be removed without significantly impacting the results.

TL;DR

It's easy to make the Joel test a better gauge of how good a company is. It's hard to do it without changing its original intent.

Answer 7

What type of weighting? For example: I can work without the best tools if I at least like being at work. On the other hand, If I have the absolute best stuff money can buy but have to work in a dungeon, code quality will suffer over time.

In my opinion, those 12 points are all equally weighted, because you can make compromises. I.e. Source control is normally an absolute MUST HAVE, but it is still possible to work without if the rest is good. On the other hand: Companies that do not have Source Control usually fail at all the other points as well.

Answer 8

I think Jason Baker makes good points that the Joel Test is about good software development practices, not whether or not the company is a good place to work. And keeping the test short and simple is best.

The only thing that I think is lacking on the test is something about recognizing non-coding work. I've found in several of my past jobs that the management expects engineering documentation and tests to be written, but doesn't allocate any time to it. I guess they either think that work takes zero time, or else they expect developers love doing it so much that they'll do it on their own time.

So I'd add this question:

• Do you count writing documentation and tests as part of software development?

I agree other practices are certainly beneficial to software development, such as code reviews, pair programming, TDD, keeping abreast of industry trends, using an open-source model, exploring new technologies, etc. But these are not deal-breakers.

Answer 9

"hallway usability testing" seems like the odd one out to me. It seems more to me like Joel's particular way of doing things than widely-used best practice, which the rest more-or-less are. There must be other ways of testing usability.

Answer 10

General questions

(1) Does the company provide the equipment and necessary software to do the development? (I worked for a company where only the more senior people had company equipment furnished - everyone else had to buy their own.)

(2) If you move from project to project do benefits such as medical/dental and 401K continue? (Eligibility was also position based, so if you moved around you may or may not have been eligible.)

(3) Does the company reimburse for travel expenses? (The PM decided to save on project funds to only half the group got reimbursed - a bad situation to be in once you find out after you get back from TDY.)

(4) Am I allowed to contribute towards a 401K, and if so, when? (I had to deal with a company where the employee could not contribute towards a 401K, but the company opened an account at Vanguard and made the corporate contributions.)

If it's a government contractor

(5) Is there money for clearances? (I've seen where to cut expenses a company decided to lay people off when their security investigation came due.)

(6) When does the contract end? And if the company is a subcontractor, when does the contract with the prime contractor end?

Project Managers

(7) How do project managers determine the metrics for raises? (One PM I worked for decided it should be on number of SLOC written.)

(8) How do PMs decides the development language? (I worked for a PM who used a blog and decided on PHP. Four months later he read another blog on how much better Python was, so he decided to dump the code base and have us start again.)

Yes, I've been through all of this and more.

Answer 11

Already posted as comment to the question, but I think this is more visible.

Question:

• Does the organization encourage communication between workers/developers/departments?

Rationale:

• An organization which does not will suffer from work duplication, social starvation and general poor motivation of the workforce.

Answer 12

I approve of the Joel test, but I think it can be improved by going away from yes/no questions.

Joel says (on http://www.joelonsoftware.com/articles/fog0000000043.html):

it's easy to get a quick yes or no to each question. You don't have to figure out lines-of-code-per-day or average-bugs-per-inflection-point.

That's fine and dandy, but I think you can get a better precision of "how much yes" the answer really is by making the question more open. Also, assume the answer is yes; it shows you expect high standards. I suggest asking something like the following:

1. Which source control system do you use?
2. Which bug tracking software do you use?
3. How do you test your software?
4. How do you schedule development? How up-to-date is the most recent schedule?
5. How detailed is the spec?
6. Let's say I know I'll need a particular book, IDE, keyboard, or chair for my work; how do I go about getting the tools I need?
7. What do you think about the idea of trying to fix bugs before writing new code? Does the organization do this?
8. How do you do usability testing?
9. How are candidates' programming skills evaluated?
10. Suppose we were to ship right now; how does the human involvement between "svn co" and Golden Master look like?
11. How do you do automated and periodic builds?
12. Which working conditions are you trying to provide your programmers with?

That way, you can always evaluate the 12 points by producing your own yes-or-no answers depending on whether you're satisfied with what you hear. But you can also get a deeper insight into how well stuff works.

Answer 13

The Joel Test is really just a starting point, a checklist.

Anything more involved moves away from that scope.

For example, the Hallway test. Joel says 5 users will highlight 95% of the problems. Not true. The original research says 85% of problems. That research is also 6 years old and technology has changed. Also, how long does a user test for?

Like I said, the test is just a starting point. If you want to improve it, research each question in its own right & develop a more effective approach. There is plenty info on each of these already.

So don't read some much or place so much faith in this test.

So to improve on the test...use something else.

Answer 14

I think the best thing about the joel test is that it is concrete facts that usually indicate problems in a development firm.

In an interview where candidates and companies are presenting their best face, it's nice to have simple yes or no factual questions that can indicate if there are problems with their development process.

I would second AtariPete's suggustion and add to the list.

"Does your company do code peer reveiws on a regular basis"

Answer 15

"Do you use the best tools money can buy?"

Sometimes the best tool is free or open source. I'd substitute:

"Do you use the best tools available?"

Answer 16

IT Questions:

• Do programmers have admin privileges to their development machines?
• Do you allow your programmers to have unfiltered access to the internet?
• How responsive/fast is your internet connection during working hours?
• Are development machines segregated from the rest of the network?
• What is the uptime of your subversion server excluding scheduled maintenance?
• Does IT support development software?