How to weed out the bad programmers from the competent ones in the interview process

Question

I am getting ready to add another developer to my team and I want to try and fix the mistakes I made in my last hiring cycle.

I like to think of myself as a competent programmer (I can be given a project, I can deliver on that project and the deliverable work with very few if any bugs) and so I ask questions that I would ask myself in an interview. I've come to the conclusion that my interviewing skills are completely lacking because the last two people I've hired interviewed incredibly well but have been less than ideal at the tasks that they've been given. My CTO (who was completely useless in giving any guidance as to how) suggested I improve on my interviewing skills.

The question is this - How does one programmer interview another programmer and get an understanding of the other programmer's abilities?

Edit: Though slightly different, the answers provided to this question could be of use to you. That question concerns specific interview questions while yours seems to be more general about interview approaches and not just about the questions themselves.

Update: Just for the hell of it I asked two of the guys I worked with if they could do FizzBuzz. 45 minutes and 80 minutes to work it out. And these aren't bottom level guys either.

Answer 1

As a top-level weeding mechanism, FizzBuzz works.

Answer 2

You are going to be 6 months out regardless of anything you do... at least that is my experience.

Have someone knowledgeable sit in with you... the more opinions the better...

Answer 3

Practical examination. Give them a simple function to write in the language of their choice and see if they can communicate to you about the choices they made in it. Pick an example project of a kind that you do, and that they claim to know how to do, and have them walk through in detail how they'd start it.

Answer 4

As for technical abilities, let the person write some code and review it. They can take the assignment home and send the results back later or do the assignment at a computer in the office, ask them to complete the code as they would on a normal day's work. Besides proper functioning of the code, design and completeness, look at things like documentation, comments and unit tests, if provided.

Nothing beats real proof of coding skills.

Answer 5

Pick some code from project euler and ask them to order them in complexity... pick like 7 ranging from problem 10 - 70 and tell them to explain their answer!

(I;ve never done this but i think it would be a good idea and i think it would tell more than the FizzBuzz)

Answer 6

Ask general questions that are open-ended. This will allow you to see that the person is capable of critical thinking and problem solving. Specific technical questions are good but in my experience they never really show someone's ability as a developer.

Show them some broken code and ask how they would fix it. Don't make it a simple compilation error, rather show them some code that has duplicated functions and ask them how they would refactor it to improve reusability.

Give them a small function to write with simple input/output and let them use whatever language and implementation they like. Then, once they are finished, change the requirements and have them refactor to the new requirements and see how they handle that.

Don't ask questions with specific right or wrong answers as they do not tell you anything about someones abilities.

Answer 7

Sit him (or her) down at your machine, and give him a some of your tasks to do. Allow him to ask you questions and for advise. You'll find out his thought process, as well as his technical ability / limitations.

Answer 8

I'd recommend Joel Spolsky's article on intervewing.

http://www.joelonsoftware.com/articles/GuerrillaInterviewing3.html

Answer 9

Read up on some stuff by Joel Spolsky

http://www.joelonsoftware.com/articles/FindingGreatDevelopers.html

Answer 10

Have the person do a code review in the interview. This is a difficult approach, as it's hard to create a piece of code that can be reviewed out of the context of a project. Perhaps take an isolated library-style method from your project that had some problems or could have been written better. Failing that, look to an open-source project for your example.

The goals here are (1) to see that the person can critique logical structure (as long as you're not hitting a Java programmer with LISP, they don't need to know the language), and (2) they're not scared to point out problems.

Answer 11

Come up with a very simple concept of the type of project you usually work on, and code it together on a white board. (I got this from a Joel Spolsky article).

I ussually do CRUD heavy business apps, so what I did last year when I hired somebody was to create a simple user-role security app. I had the interviewee create a data model for all the information we might need, then a class hierchy, then UI, etc....

We'd discuss trade offs and alternate ways to do things constantly and actually write code on the whiteboard. It gave me tremendous insight at how good these people were.

I was suprised at how many people interviewed well, and fell apart at the whiteboard.

Answer 12

Irrespective of what questions you end up asking, as I'd suggested in this question here, I would recommend having a standard set of interview questions. That way you are easily able to compare one candidate's abilities with those of another.

Answer 13

I would seriously consider prioritizing candidates with open source contributions. If they are part of a team, you know that at some level their work has been vetted by other competent people at some point along the chain. If they were the sole contributor to their program, then you at least have a relatively large code base to look at. The software I wrote for my graduate work got me several interviews and a very good job. Additionally, it knocked out all of the "but can he code" questions. It left us to discuss what the job entailed and salary parameters. An additional benefit is that you can look at their user name on the forums (assuming it is the same as the svn/cvs name) and see how they interact with other people. How one interacts with others on a forum is a window into their true person. Someone who is kind and giving while anonymous is likely to be the same as an employee. Someone who is a jerk may charm you in the interview, but their true nature will eventually come out.

Answer 14

I am not a manager, but I have done programmer on programmer interviews.
Mix it up a bit.
Ask them to pseudo code an example of a simple problem. Language specific may be OK if you can provide them a computer. Pseudo code if on paper. This is more of a logic test than language test.

Ask textbook questions. These aren't necessarily used to determine intelligence on the subject, but more to shake them up a bit. Here is where you watch for the "baffle them with BS". A lot of people can't remember textbook answers from college, those you want will tell you if they can't remember, but will be able to tell you where they can get the answer. However, a dismissive "I'd Google it" is not the best answer, although it is a common one.

Take off your programming hat while talking to them. If you don't think a junior programmer will understand what they are talking about, ask them to explain better. Some people speak technical jargon, but don't understand it. If they can't explain it better...

Pay attention to their attitude. Cocky is OK to a point. Nervous is Ok to a point. Were they punctual to their appointment? Do they look at their watch a lot?

Spend a couple minutes talking nonstop about the company, parking, perks, workload and such. Do their eyes start to wander while you are talking? You may be boring, but you deserve their attention if they are serious about working there.

Take them out to the floor where they may be working. Do they appear interested in what they see? Look for grimaces when they see the cramped cubes or messy work areas. Look for interest when they walk past a programmer actually working.

All the above is subjective and not one will make me hire or not hire anybody. Just some simple things I do.

EDIT... I forgot this important item. If they say something you don't understand (may not even be programming related), question it. Never be afraid to admit you don't understand what they said. If it's BS, it'll show. But if they can mentor you on something in the interview, they will be able to mentor others on the floor. This is a great skillset to be looking for also.

Answer 15

You need to look for two things: Are they smart, and can they get things done? You often see one or the other, but when you see both, grab that candidate. A distant third is: Can you work with the person everyday?

I got this advice from Joel Spolsky, and it worked brilliantly. It's a simple matter of knowing what to look for.

Answer 16

What other people said here is right. My 2 cents: In countries where people usually have computers and internet access at home (as opposed to India and similar), ask them about coding that they have done at home. I don't think I know even one good programmer who doesn't code for fun!

Answer 17

I think the first step is to decide on your criteria. For example, rate the following criteria in descending importance for you and your team/organisation. This process enables you to decide what's important to measure during the interview.

• Smart - how intelligent is the candidate?
• Drive - how good is the candidate at getting things done?
• Motivation - what is the candidate passionate about?
• Curiousity - how interested is the candidate in his/her craft?
• Skills - how skilled is the candidate in what s/he is going to be doing?
• Knowledge - how knowledgeable is the candidate in the technical and business domains?
• Communication - how good is the candidate at verbal and written communication?
• Social - how well will the candidate fit into your team and organisation?
• Education - how well-educated is the candidate?
• Ethics - how honest and trustworthy is the candidate?

The next step is to decide how you're going to measure candidates against each of your important criteria. This is the creative part of interviewing. You can google for many different measurement approaches, pick your initial favourites, and then adapt over time.

The next step is to write down all of your interview questions, so that you run every interview in a consistent way, This enables you to measure how well your interviews are working, and hopefully make them more effective over time.

The final step is to write notes and conclusions about every interview. Again, this helps you to improve your interview process over time.

Answer 18

The bad ones are bad because they have no clue about what they're doing, they're not passionate about the technology they produce and have a bad attitude.

I try to get people to talk about the stuff they claim to know - their area of expertise; even if it's not really relevant for the position. Then I drill down and challenge them with questions in order to see exactly how much they really know. The good ones can explain the material well and thoroughly to someone (like me) who isn't necessarily an expert in that particular field. Next, I try to push further - beyond their knowledge - to see how they react and whether or not they admit to not knowing something (mostly people are FOS and just try to BS their way with made-up answers).

People who are passionate about what they do, can explain themselves to other people well, and have no problem admitting when they're lost are probably good candidates.

Answer 19

Some good answers to this on "How Do I hire a programmer smarter than me?" This is what I had to say:

If I were hiring now I would start with Steve Yegge's phone screen questions because most people who apply for jobs appear to be clueless liggers. If someone came back with "hey, you're just repeating Steve Yegge's phone screen questions" that would pretty much guarantee them an interview because it would indicate they had read something from that direction.

Finding things like what books, blogs and websites on programming they used a lot would be a good start. If they read any that might indicate that they are in the in the top percentile of programmers who are still working to improve their craft. If the books or sites they favour are more along the lines of The Pragmatic Programmer than SQL Server For Dummies you are probably on the right track again.

Better still if they have their own blog, or other public profile to indicate their expertise. If, for example, they had a profile on this site that showed a trail of insightful or accurate responses that would be another good sign.

None of these things guarantee expertise, but they do indicate the kind of mindset that I would like to see in a programmer I work with.

There are also warning signs I look for- this is purely personal belief, but I don't think a programmer who only uses one language can be as good as someone with equivalent experience who has used several. Each programming language you use gives you new insights and techniques that you can apply to all the others you are utilising and the type of person who is not interested in trying new languages or experimenting with new technologies may not be ideal for you if you are creating a new application.

I think if someone is really passionate about what they do it that will come across when you talk to them about it and someone who is passionate is going to be more motivated and probably do a better job as a result.

Answer 20

There's a book called Programming Interviews Exposed which takes on this problem from the potential employee perspective. It exposes technical and general questions so it should be a helpful guide even if you're the interviewer.

Answer 21

My boss is quite good at this, because (quote) he made enough mistakes.

He always goes for:

1. Does his personality fit the team. Casual talk about various things work best, because people tend to loosen up and show who they really are. My boss is always stressing out that working atmosphere is more important than individual competence.
2. Is he able to think methodically. Give him a task/problem and see how he approaches the task.
3. Can he get things done? Does he have self-management? This one is hard to find out in a interview.
4. How does he react/perform under pressure.
5. Is he able to make himself understood? Even to a person with no background in the subject.
6. Is he able to apply his knowledge in changing situations. (I don't know how to translate this sentence. It basically means: Can he apply his knowledge in a completely different environment/situation)

if the candidate manages to fulfill these criteria, he starts with the field of work questions.

My boss does not work the list from 1 to 6. He always mixes the question and asks personal questions (very casually) in between the job related questions. He does it this way so people can't adjust. This way the answers are more honest. This might sound very confusing, but it actually works.

Answer 22

The question is this - How does one programmer interview another programmer and get an understanding of the other programmer's abilities?

Team interviewing to a pre-agreed set of criteria based on the job description is the best way to avoid the pitfalls you have described.

The problem you face is that, as a programmer, you have areas of interest. Should a potential hire touch upon one of these, you will automatically begin to prefer them - whether or not they have any actual skills.

The best interviewing candidates can mask a plethora of flaws by carefully digging to find your "weak spots" (i.e. the stuff you love) and then focusing the entire interview on these. I've seen interviews totally hijacked because the candidate and one of the team absolutely loved some recent movie or TV show.

With the team approach, you have a better chance of staying on track. Also, while one of the team might be dazzled by some facet of the candidate's background, others may not - and so the overall counterbalance effect serves to provide a much better debriefing of the interview.

If you are not interviewing using a team, then you will certainly be hijacked by a classy candidate who happens to share your interests - whether or not they can actually do the job.

Cheers,

-R

Answer 23

Put them in a 30 foot high steel cage with one chair, one desk, and a computer with your compiler and editor of choice.

Throw in the specs for a program

Lock the cage.

See what happens :)

Answer 24

I think good programmers love their craft.

Try to get the person to express opinions on things related to their craft. If they express opinions that differ from yours and you can get a good discussion out of it I'd say they are prime candidates for the job.

Also as previously suggested try to find out how they approach improving their skills. Bad developers generally don't while good developers read a lot and experiment with stuff at home.

Answer 25

I went through this in my last round of developer hiring with an Indian outsourcing firm where we got to pick from their pool of programmers. Having them write sample code (simple projects to demonstrate several concepts that you really will encounter in your line of business, no more than 1 hr to complete) was INVALUABLE. I had 3-4 different samples since we were interviewing developers that potentially had access to previous failures.

Answer 26

I know this is already a Joel Spolsky love-fest, but check out his book Smart and Gets Things Done, it is most excellent.

Text book answers suck because they're too easy to game. I've met several folks who were great at all of the standard puzzle questions but would never be able to ship a product on time. That said, you should have a minimum level of weeder questions -- the FizzBuzz problem is great -- to get rid of the first 90% of people who shouldn't be applying in the first place.

Ask people about their past projects. Find out what they're excited about. What projects sucked for them? What went wrong? What was their involvement?

I think one major mistake that employers make is focusing on just the technical talent. While the technical talent is certainly a prerequisite, what you're really looking for is someone who can deliver.

Think about it. Have you ever been in a situation with a rockstar programmer who refuses to implement a feature because he doesn't "believe in it" or thinks the customer is "stupid"? Unless you're working for a non-profit or an open source project, you're in the real world and someone is paying your company good money to have a product delivered. Your greatest resource is someone who can get a good grasp of your customer's problem and deliver a solution that works, on time and on budget. It sounds so simple because it really is -- you would be surprised how many candidates just don't get this.

I agree with others here about getting the candidate to fix broken code. I'd go further and ask them to design a system from the ground up. How would they design a complex system? Do they have good, reasonable ideas? Do they have janky, crackpot ideas?

Just remember: this guy is going to be responsible for a part of your business and may be involved with your projects. Do you trust them to do a good job? If the answer is maybe, don't hire. Never hire the maybes.

Answer 27

There's only one way to find out if a programmer is any good in an interview:

Ask them to write some code. Asking them to solve a basic, common business need your team regularly has to deal with is one of the best ways of gauging if they are competent or not.

Answer 28

My contribution:

Thirteen Patterns of Programming Interviews

Answer 29

One thing that really separates the wheat from the chaff is extracurricular activities. Ask them what programming-related blogs or websites they read, if they've attended any conferences lately, or what kind of projects they work on in their spare time.

It's a real quick way to determine if someone has an interest in the subject of programming beyond the paycheck.

Answer 30

Before you start, read Joel's excellent Guerilla Guide to Interviewing. I particularly like the observation that you are looking for people who are:

1. Smart, and
2. Get things done

I have interviewed no end of people who can only do one of those two.

It's vital to make sure that people can code. For C++ jobs, we make them solve simple programming problems, on a whiteboard, in front of us. We then talk with the candidate about:

• bugs in the code (there are almost always some)
• boundary conditions
• performance
• style
• const correctness
• etc.

I am looking for someone who can understand the problem, write code that will work, understand and react positively to constructive criticism and (most importantly) defend the way they have solved the problem.

Examples of the coding challenges might be:

• Write a function that returns the area of a circle given the radius
• Write a function that returns whether a string starts with an upper case letter A-Z
• Write a function that adds up all the values in an array
• Write a function that converts a string to upper case
• Write a function that strips white space from a null-terminated string in-place

The problems are deliberately vague - we award bonus points for candidates:

• Asking about the types of input (or even better using templates)
• Considering unicode or wide character sets
• Making their functions const correct
• Using the STL or other standard libraries

I've been amazed at how many people with all sorts of "technologies" on their CV go to pieces at the second problem!

Answer 31

I think a FizzBuzz type problem is a great way to see the candidate in action. People do get the jitters in interviews, but if you don't ask the candidate to actually write code then you haven't really interviewed them.

I try to use the same basic outline for all programmer interviews, so that I can fairly compare candidates:

1) Go over the candidates resume, have him or her answer basic questions about the projects they've participated and the companies that they've worked with. Try to get a feel if they've work in an environment similar to yours. For example, if you're in a startup with no rules and the candidate came from IBM or the DoD then you need to figure out if they really would be happy in a less structured environment. Conversely, if you are in a large organization with lots of rules and procedures then a programmer from a 10 person startup might not be happy submitting to your management hierarchy.

Try to get a feel for the actual work the candidate did. This is a good time for one or two 'tell me about a time' questions, i.e. 'tell me about a time you had to quickly solve a hard problem under pressure'.

Ask 'what are your weak points?'. If he or she says anything other than 'I like to hit people who ask really stupid questions in interviews' then end the interview there.

[ETA] I always ask for candidates to bring sample code, this is where I have them show and explain their code to me. The best candidates will actually whip out a laptop and demo something.

2) Have a list of the skill sets that are important in the position you are filling. Ask the candidate to rate his skill in each area from one to ten, with one being 'never heard of it' and ten being 'I invented it, or wrote the seminal book on it'. What you're looking for is honesty and logical consistency. The absolute values of the answers aren't important except for the zeros, ones, nines and tens. An excellent programmer might not be familiar with some tool or technique you use, but if he or she is honest and smart they'll come up to speed quickly.

3) Have a list of technical short answer questions on each skill area you asked about. Don't waste time asking questions about the subjects the candidate self-rated as low, drill into the areas where he or she claims expertise. Try to think of questions that come up in real world programing that are typically glossed over in academic study.

4) Think of a design and implemenation problem you've had to solve. Ask the candidate how they'd handle it. Since I do mostly desktop applications, the question I like is 'You need to add an application update function to a desktop application. How do you implement it?' I'm hoping he'll describe an updater that includes security and reliability considerations. You're looking for signs that she's had experience in solving design and implementation problems in the real world.

4) Give the candidate some programing problems, starting with something like FizzBuzz, followed (if he or she can do that) by more difficult problems until you realize you've passed the point of diminishing returns. This is also your best chance to see if the candidate is someone who's personality will fit into your team, something I consider of primary importance.

If you have more than one interviewer then these steps can be parceled out.

Good luck.

Answer 32

I think the why behind the answers is crucial. Do they understand why a certain thing works certain way. For example all candidates will tell you that a function cannot be overloaded by return type you will be surprised how few will be able to tell you why and where is the ambiguity for the compiler. With the plethora of interview question sites most folks will know the answer and will be able to parrot it back to you, the imporatnt thing is whether they understand the rationale behind the it.

Answer 33

Give them a -correct- (but ugly) piece of code to rewrite. Ask your current developers to provide sections of your actual code where they had found some tricky bugs, with these bugs reintroduced.

By no means request writing programs on paper. Even very good programmers get confused by two-three "typos" and unable to edit them out make a total mess of the program. Give them a computer not connected to the net, maybe nothing more than Notepad.exe and tell them to use it.

Answer 34

Candidates definitely have to write some code, it doesn't have to compile, but you can tell if they know how to program or not.

Also, I strongly recommend using short 5-10 minute phone interviews as part of the process. This will help weed out people early on.

Another time-saving tip is to have a non-expensive staff member do the phone interviews (that is, not the project manager who's on $100k a year, use the office administrator or contract someone temporarily.)

I have some more information about the interview techniques I use here: Hiring Programming Staff

I hope that helps.