Add a programming quiz to the job advertisement?

Question

At our company, we are looking for good Java programmers, that the one and only requirement and that's written in our job advertisement. Since we need more people to handle all the work that's coming up, within this and the next year, we want to hire people. The problem is, most of the candidates are not as good as we want them to be. Maybe some of them think they are good and simply underestimate the task. Maybe some know that they are not the best, but try it anyway, because all the job advertisements read like everybody is looking for a 25 year old übergeek with ten years work experience and a master in computer engineering. However, 80 - 90 % of the interviews are deflating and we can't offer them a job.

The question is: how do we make clear, we are seriously looking for some good programmer? If you not understood OO or not really know how synchronized works, you the wrong man for the job. In a nutshell: how do we increase the quality of the applications/candidates?

I suggested to include one or two programming questions in the advertisement, but my colleagues don't think this would be a good idea. In fact, Joel also don't think it is, as you can read here

I’ve frequently heard the suggestion of including a programming quiz of some sort in the application procedure. This does work, in the sense that it reduces the number of applications you get, but it doesn’t work, in the sense that it doesn’t really improve the quality.

I'm not sure if Joel means the same thing I do. I don't want the candidate to send the answers in, I just want to show him what he have to know to get the job. He should decide for himself. I think this is more precise than arguing about how good a programmer is. Otherwise, it is like asking How would you rank your java skills from 1 - 10 instead of doing a test.

So, what do you guy think? Should one add a programming quiz to the job advertisement? Why / Why not?

Answer 1

I strongly suggest you do.

In the past we were looking for a Java Web programmer and asked for all applicants to send a simple deployable java web application implementing a plain chat room (no security, no users, no persistence - everything is in memory, no ui design).

We didn't even check whay they sent (as long as it was a .war file). That greatly reduced the number of applicants. But yes, after that you have to start asking questions, calling references, checking web profiles, etc. But it was only two dozens of candidates relevant, not > 100 which is difficult to handle.

Answer 2

WHY: weeding out the wannabes is always good, so YES, include the programming quiz

WHY NOT: being a good programmer (passionate, thorough, smart, consistent) and being good at an interview (keeping your cool, thinking fast when under pressure) are two separate things, so include some simple quizzes, not tricky questions (like the Java certifications ones, to make myself clear)

Answer 3

Spend five minutes per promising CV and hold a mini-phone interview in which you ask a few interesting questions, such as pros and cons of encapsulation or whether strong typing is always indicated.

It's been my experience that CVs bear almost no relation to the quality of the applicant, but it's near impossible to fake know-how in spontaneous conversation.

Yep, you'll maybe lose some good candidates who lack confidence (depending on your own manner of course) but this is a small price and a small investment in time with a big pay off.

If you can't do it personally, delegate to a developer you trust. Do not delegate to an HR person.

Update: to answer your question - NO, don't put a quiz in the ad; it's too easy to game just like CVs.

Answer 4

Why not? If it is just another form of specifying a qualification level, as you describe, it is definitely better than many other, more standard ways of doing this.

Please, provide a link to your job advertisement, I might be interested in applying. :)

Answer 5

Like all decision problems, you have to understand that this quiz has two problems: It can reject qualified candidates (possibly by discouraging them), and it can fail to reject unqualified candidates. The harder you make the test, the more you will from false positives to false negatives.

Now, the same theoretical objection exists against any kind of selection mechanism used in an interview. Therefore, the proper measure is qualitative. What kind of error are you willing to accept in either direction, and are the better selection criteria available?

The big advantage of using a standardized quiz is that it's repeatable. This allows you to improve on it, as you conduct more interviews. With impromptu questions, it's harder to improve your selection skills.

The explanation quotes Joel, in particular to state that extra hoops are bad when it comes to gathering resumes. It's not a bad idea to state in the resume-gathering phase what you'll be looking for during the interviews, by giving examples of interview questions. That is not an extra hoop. Instead, it shows you are running an open company, and you hire based on fair&reasonable criteria. People can exclude themselves, and if they do react you can ask their opinion about the example questions.

Answer 6

You should not add a quiz, because

1. really good people have many options and they may decide not to apply just because they feel lazy filling a quiz for a company which hasn't offered them anything yet and

2. not so good people will find answers anyway and send them to you and you'll have to decide whether or not you're satisfied just to invite them and see they don't fit immediately.

You shouldn't overstate that you want really good people. Really good people are often self-critics and they may decide they are not good enough for a "super ultra hardcore Java development position" they would really fit well. Not so good people are often overly confident or apply just for any job. So you'll scare the best and not get rid of the worst and mediocre.

Answer 7

I would say definitely don't add a quiz, at least not as a mandatory requirement for applying.

As a job seeker, the thing I hate the most is the investment of time in preparing cover page and application materials, and then not hearing anything back (not even a negative). Unless your tests automatically tells the person whether he qualifies for the next step, you are just going to aggravate people who spent too long on the test with zero feedback. If you can't make the commitment to get back to everyone, don't ask someone to spend the time to do the test.

In addition, if my time is a scarce resource, I would much rather try ten places that would take me 10 minutes each and maybe get one contact, then spend 100 minutes at a company where I don't even know if my CV would be of interest to them.

Besides, being able to pass a programming test doesn't mean they meet your needs. Let's say you are seeking an experienced J2EE developer with five years experience, an M.S., and good SQL development skills. How are you going to filter out the high school graduate who played some with beans and coded in Access? It might not save you that much time.

Code tests are an important step before an onsite interview, maybe even before a phone interview. But by making them a requirement for application, you will be sending away many unqualified people who feel it might not be worth the investment without clear prospects of return. In other words, if I dreamed of working for company X and that would be the requirement, ok. But if you're just some other Y and you make that demand upfront, you're either presenting yourself as more elitist than you are, or you're not competing with the market.

I can tell you of several companies I considered applying to, but didn't bother because of the automated pre-application code test. Read my CV, talk to me, and then I'll do whatever tests you'd like. But give me the basic respect, rather than a Captcha.

That being said, feel free to write sample interview questions that you no longer use to scare away people who can't answer them. And regularly google your actual questions if you do put a test to make sure nobody blogged the solutions.

Answer 8

Don't add a quiz.

Change the job advertisement to look for a 'Senior Java Developer'. People with less than 5 years experience will typically shy away from applying.