What about fixing actual bugs in an interview?

Question

My primary responsibility is a software developer, but I have also been tasked with doing the first level technical interviews of perspective clients. I enjoy it, so I am glad to do it, but I am finding it challenging to objectively screen candidates. I have a list of programming "trivia" questions and even some programming challenges they write via live meeting, but I want to take it to the next level.

What I am thinking is I could select a really simple (not just form typo) bug in our actual code base and then had them fix it via Live Meeting. Of course they would need to ask questions, and I would be there for that, but that would also be part of the interview, do they know what kind of questions to ask, etc.

Downsides:

1. Intellectual property issues: I don't think there is too much of a threat of our code being stolen, but our lawyers might want an NDA before we can interview someone . . . .
2. High domain knowledge needed: Less of an issue because I would be there to explain things as they needed them.
3. Might take a couple hours: True . . . but still cheaper then flying them in for an interview and putting them up for the night.
4. Code might scare them off: Showing a candidate a legacy code base might be scary, although it would be better to scare them off before we hire them.
5. Code ownership issues: Could I actually use the code they wrote in an interview in a production application without compensating them?

Upsides:

1. As real world as it gets: This is almost like hiring them on a trial bases, but without the pay.
2. Kill two birds with one stone: I screen a candidate and I get one of my bugs fixed, of course this assumes they can fix the bug . . .
3. Tests for exactly what we are looking for: The ability to work in our existing code base is what we want, so lets make sure they can do it.
4. Makes coming up with interview materials trivial: There always seems to be bug reports and feature requests. No shortage there.

The intellectual property issues might be a deal breaker. I would obviously need to get it cleared by management and legal. I am curious what everyone things from a programmer point of view? What that scare you off if it happened during an interview?

NOTE: If you think this is a TERRIBLE idea, then please leave an answer (or vote one up) that says that. That is why I am asking, to find out if people thing it is a good idea.

Answer 1

I think the major downside is that any developer with any nous who you ask to fix your bugs for free will tell you to p*ss off.

Answer 2

I'd feel a bit used if I was given that sort of task in an interview, for precisely the "kill two birds with one stone" upside.

Answer 3

I think you're treading on thin ice, ethically speaking. Someone interviewing for your company isn't getting paid to fix defects in your company's codebase. Though they will likely want to display their competence and ability, I doubt many people would want their thoughts/ideas to be capitalized upon without compensation.

All said and done, I certainly wouldn't appreciate my own interview to be taken advantage of in such a fashion. Yes I want to prove myself in their environment, but not like this.

Edit: why not have them work on fixing "defects" in a deprecated and unsupported project? I'm sure you have code sitting around which is no longer supported, and you could find bug fixes in previous versions which you could ask them to solve. This would also allow you to compare their answer to one which was implemented by your company.

Answer 4

Your biggest problem is that an interviewee won't appreciate a potential employer just using him to fix their bugs. It just reeks of a cheap trick to get free labour, whether its a valid interview or not.

Answer 5

I like the theory of the idea, but in practice usually to effectively fix a bug you have to know the code base you are working with. Right there as a developer I would end up spending a lot of time just trying to understand whats already there before I could begin thinking about how to fix a bug in the system. One of the first things any new developer needs to do is spend time learning the code base, how it works, coding styles and conventions used, etc. I think it may be more applicable to ask the developer being interviewed to show you a bug they fixed and to walk you through the process they took to fix it.

Answer 6

The idea sounds interesting.
The potential downside I see is showing / explaining the candidate enough code to give him a chance to fix the bug. This might be very difficult except for a few bugs chosen carefully.

Also, you will need to look into the resolved bugs bucket as well for the above reason and also because managemet would not want you to sleep over a bug to save it as an interview question :).

Answer 7

I would probably create a bug in the software for the interviewee to fix, it doesn't seem right for you to get them to resolve a bug for you for free. Also, you can control the level of difficulty.

Answer 8

I love the idea. I'm not comfortable chatting with people I don't know about computers. I am always hesitant to use technical language because I do not fully know their background. This way I would be in my own environment and feel comfortable speaking about the code directly instead of choosing which thoughts I have to explain to laymen.

Answer 9

Checking people's ability to find a bug is one thing, having them do it as a contribution to your project is something else. And if you yourself do not know where the bug is then something that appears trivial may be quite in-depth.

It also sounds impractical. When I have to fix a bug in unfamiliar code I will often examine the program running in the debugger to get a feel for the structure and validate any assumptions I may have made. Unless you're also going to provide the candidate with a set of docs, UML diagrams, and a couple of hours I don't see how else they would approach this. Static analysis can be a very difficult way to ascertain the behavior of a sizeable system.

I would suggest a similar but perhaps more sound approach would be to develop a simple sample app that's inline with the domain of your business. Rather than (or as well as) fixing a bug you could have the programmer implement a new feature. This will not only show you how well they can grok a codebase, but how they choose to extend it and integrate new functionality.

This test can also be something the candidate can do as homework since it has little reliance on existing knowledge, and something you or your team can mark at your leisure.

I would also save these types of questions for a second phone screen. Your initial phone screen should be quite capable of weeding out poor candidates with FizzBuzz type questions.

Answer 10

Regarding upside #2: if you don't already know how to fix the bug before giving it to the candidate, you won't know if they've actually fixed it...

Answer 11

I don't see this as a good idea primarily because your interview process becomes nondeterministic. While your impressions of a prospective candidate are always going to be subjective, your interviewing methods should not.

In the past I have been walked through artificially planted bugs in an existing legacy system to test knowledge of tools, language semantics, etc., but not to fix actual bugs in a deployed system.

Answer 12

NDAs are not unheard of for interviews, but generally not until you're actually bringing them in.

It doesn't feel right or fair to me to have them 'fix' a current open bug in your database, but I think it's probably not a bad idea (subject to several of your other potential issues) to have them go over a bug that has already been fixed (this way you can know if it really an appropriate-sized problem).

Answer 13

It is offensive.

I even felt offended just by the question lol :(. Don't take me wrong, I see a case for it, but if I feel this way just about the question I would almost certainly walk away from the interview.

Also consider the effect in the company's image when it starts going all over in twitter/[insert social media in here] :P.

Answer 14

Perhaps a better approach would be to take an already fixed bug from your bug tracking software and give the prospective employee the details as they were entered into the ticket. Then ask them what they would do to track down that bug if it were their first assignment.

Ideally, you'd be driving not so much at a technical solution as determining their problem solving skills. If they are to be an employee for a long time, chances are good they'll end up using more than one programming language and the ability to problem solve comes first, then how to implement that solution in a given language.

Answer 15

My answer if I got this question would be: "Where is your failing unit test please?" ;-)

That would tell me as much about your company and its development process as what it would tell you about my ability to fix your bug.

Double edged sword indeed!

Answer 16

It feels a little exploitative to me - like you're interviewing people just to get bugs fixed for free. I know that's not what you're doing, but if it happened to me, I might be suspicious.

What if instead of having them fix an actual bug, you took a bug that is typical of the codebase and simplified it a little? It would be a neat way to see how different people approach the same problem and you'd probably figure out pretty quickly what the "good" people do with it.

Side note: I've never done an interview that didn't require an NDA. How could I ask questions about the company, the technology stack, etc., without one?

Answer 17

Why not adapt this a little bit? Take a bug that you've fixed in your code base, and write a small application that has the same bug, in the same style as your own codebase. This will take some effort from your point of view, but not much in the way of time. Ideally, you're talking about reducing the bug to the least amount of code that showcases it. In fact, come up with several scenarios with various bugs that you've fixed. The wonder of version control is that you can use it as a reference to do things just like that.

Now, then, come up with 5 or 10 such examples, and give one to an interviewee. Change 'em up over time, and add to the pool of examples, such that you don't have a predictable interviewing environment, because you can count on at least some of the interviewees to turn around and go to something like Stack Overflow or Yahoo Answers with your code snippet and pose it as a question, so you don't want your interviewees to be tainted by being able to spend 3 minutes to Google and find the fix for that particular code example.

You get the things that you want for it, and you also get around any ethical, legal, or professional concerns that would get in the way of using the actual code for the actual system. Also, you wouldn't be inappropriately benefiting from the use of someone's free time, and you can then gauge how well the person is with things like working to fix a bug. You also then don't have to explain anything to them, since the example code should be short enough that they can spend fifteen minutes or less getting to know it by reading it, studying it, and mentally executing it. It should be code that compiles, and it should be something that they can, if they choose to do so and go the extra mile, write a test case of some sort for. (I'm not saying to give them a testing framework, 'cuz those require familiarity. I'm saying make it an example that you can write a small test for, without a testing framework, and be able to prove the fix; any good programmer ought to be able to do just that, and if you use a particular testing framework you might just have a good programmer that doesn't (yet) know that particular testing framework, again tainting the result that you're looking for.)

Just my 2¢.

Answer 18

An important downside

How are you going to compare two prospects if the bugs are not exactly the same?

You may be losing one good candidate, only because you thought the bug should be solved in 3 hrs, when in fact it would take 3 weeks and the candidate couldn't do it on the interview ( he might have it done in 1 week though, because he is a very good bug fixer )

On the other side, you could hire someone that for some special and unrepeatable circumstances fix the bug but never again could do it. ( probably the bug was trivial, although in your evaluation it seems complex enough ) In this case you would be hiring a guy that by luck make it to solve a bug in 3 hrs ( when it was 30 mins bug )

If you are not using the same evaluation criteria, you won't be able to make a good choice.

If you use the same bug, you will only have 1 bug solved.

Answer 19

Your company will not own the bug fix unless the applicant has signed an "assignment of inventions (including copyright)" agreement prior to the job interview. ... which is unlikely. If it is a brilliant bug fix, and your company uses it in the product, you may create a delightful and potentially lucrative case of misappropriation for a lawyer. Query whether a job interview is enough "consideration" to support an effective grant of copyright, even if it were signed over in advance. You are asking for a work product, combined with a test to display skills. Better to go the route suggested byPeter Richards above - a test of technical skills - not a contribution to your software.

Answer 20

I don't see this as being an effective means of determining a candidate's talent. You run the risk of setting the bar too high or too low and sending the wrong messages to the candidate as a result.

They may also see you as coming across as arrogant even though you're just trying to determine their skill level. This can back-fire if you set the bar too low and the challenge is seen as being too easy by the candidate. They may loose interest and respect for you and your company (ED - given Wide Orbit is a respectable company, you wouldn't want that).

Why not just put them through a standard aptitude test instead? You'll be able to determine how the candidate will contribute to the team, learn, adapt, etc. These are vastly more important qualities than being able to bug-fix.

Answer 21

I agree with those who point out the obvious problem of it being non-deterministic, in the sense that you don't know ahead of time how big the bug is.

Taking a bug that you have already fixed, giving them the stack traces and relevant classes, and asking them to walk you through their thinking process to isolate the bug may be more useful.

Even then, though you run into the problem that you're going to end up with people who happen to have a high degree of competence in your particular technologies. That is, given this system of interviewing, a candidate who isn't that bright but happens to have worked with your technology stack will have a huge advantage over the candidate who is a much better programmer, but maybe isn't familiar with some details of your libraries.

You would have better luck screening with something like FizzBuzz which only tests core competency.