Online Developer Testing for Pre-Screening Candidates

Question

Are there are any good online developer testing sites for pre-screening candidates? I know the freelance site ELance has a "certification" system where candidates can take tests on various technologies. It won't be 100% accurate, but it at least might expose the less experienced candidates.

Answer 1

Consider Brainbench at www.brainbench.com. I used it years ago, but I hear it has only gotten better.

Answer 2

Peformance Assement Network has pre-hire assements. I don't know if they have what you are looking for, but they might have something related or can create something for you.

I just talked to someone I know in their IO psychology division, and he said that they don't have a lot of technical stuff at the moment, but they have a lot of tests dealing with cognitive aptitude dealing with engineering etc., and they are pretty good at creating new tests depending on what you need.

Okay, so here is what you are a looking for if you want to create a test to screen candidates.

1. The test has to be either proctored, or the questions have to be timed.
2. The questions have to be randomized, and a subset needs to be pulled randomly from a much larger pool.

Online testing vs the phone interview.

If you hire using an online as a sole basis of hiring, then your method of hiring is going to produce faulty results. It's one step in a many step process. Can people BS an on line test, of course they can if the test is setup incorrectly. Most people assume that taking a test online is just answering questions on a page and taking as much time as you like to do so. This isn't the case. Most pre-screening exams are either, based on determining a person's personality (meaning there is no correct answer), they measure a cognitive skill (like spatial recognition, or mechanical reasoning), or they are made in such a way that it makes it extremely difficult for a person to cheat (well the good ones at least).

In my opinion phone interviews in the first step are really a waste of my time, and here's why. I have to spend time asking a candidate questions. My time is valuable. There are a lot of other things I can be doing instead of interviewing people over the phone. If I have to interview 8 people and it takes about an hour for each, I've just lost an entire day. If none of them get past the phone interview, then I've just lost an entire and gotten nothing out of it. With online screening, I can send the person an link, or have them go to a company which proctors ezams. When they finish with the test, I get the results and I know within about 2 minutes I know whether or not I wan to continue with that person. That sure beats an hour with an individual, especially if that person isn't a good fit.

If I need to, I can send out a battery of tests to 50 individuals, and when they are done, I can get a basis for their technical ability, their personality, and their potential capabilities for a given field. Phone interviews can't do that easily. With phone interviews, you are really limited to simple questions: "How does X allocate memory." You can argue that someone could talk you through a process, but in truth that gets difficult to follow, especially considering the alternatives. With online tests, you can ask them more detailed question, have them write code and submit it etc.

After they get done with that step, does that mean their are hired, of course not. After that then then you start interviewing them. It's just filtered a number of people without spending a bunch of time.

Here's the thing about creating and using tests online. There is a real science behind creating assessments. There's a whole branch of psychology devoted to it. For someone to say they can whip up a test, because it's easy and they know exactly what they are doing is akin to someone from the account department saying "I read this book on programming over the weekend, and I don't see why what programmers do is so hard."

Update to Anthony's response

I understand your opinion, but I have millions of data points to back what I say. Most situations aren't that unique, despite what people want to believe, certain types of people work well with others. It's shown over and over again in studies, about how to predict group dynamics. Personality tests show predictors as to how people think and act.

When you talk about discovering personality disorders through conversations that tests can't predict, like which ones? Seriously a 30-60 minute conversation about a person will show next to nothing about a person especially compared to trained experts who've spent thousands of hours writing tests to predict behavior. Even the best interviewers can be fooled. Good interviewees, won't ever let you get to asking those questions that he/she doesn't want answered. Their whole job is to start the interview and control the conversation. If they are really good, you'll end up leaving the interview going "wow that person was really smart and never get to ask a single detailed question." Saying you'll know when this happens is like saying you'll always know when someone tries to con you.

So, on to your statement about fooling tests. Some tests are easy to beat, you're right, but the properly written ones aren't. Think I'm wrong; try it. Try beating a cognitive personality test, or a spacial recognition one, or a timed subject test. (Here I'll even give you a hint: When you think about answering all the questions in a certain manner to make it look like you are a real team player etc. that's a mistake, some questions are meant to be missed, or not receive the highest possible score.)

Most tests are not adaptive in this respect.

On a side note, that's actually called Computer Adaptive Testing. It's been around for a decade or more and is pretty standard now.

Now can tests be relied on 100% of the time, of course not, but they'll catch a lot, and that is what follow up interviews are for, but they will weed out a number of undesirables.

In response to Anthony's and Annakata's response, they are probably right, but this isn't a sure fire predictor as to how a company performs.

For the record, we use online testing to help hire people, and we score an 11 out of 12 on the Joel Test.

Answer 3

Online screening is a total waste of time and money. There are a number of free databases online with answers to potential testing and interview questions. Any person with reasonable Google skills and a minimal amount of development knowledge can ace any test. Also, you have no idea if the person who takes the test actually your candidate.

A smarter approach is to use a quick phone screen of approximately ten minutes with rapid-fire technical questions. With the right questions, it is very easy to determine if a person is technically competent or not. By rapid-fire, I mean you will ask 20-30 questions. The candidate is prepped at the start of the interview to understand that they should answer the question as quickly as possible and that, if they don't know the answer, it is fine. Just be disciplined and immediately move from one question to the next. Don't let the candidate B.S. Either he knows the answer or he doesn't. Additionally, I always tell the candidate up front not to be offended if I interrupt him. After about 20 seconds of any answer, I usually cut him off and move on to the next question.

Ask questions you can easily score. By the end of the phone screen you know for sure whether the person has "got it" or not.

Additionally, you can measure other items with this sort of phone screen that can't be measured with an online test. For example, you can determine how well a person can "think on their feet", how well they can communicate, and how they react to pressure.

The questions I ask are generally relatively difficult--as my philosophy is ONLY to hire the very very best people I can find. I always ask at least one or two questions I think no one will be able to answer to see how people react. (There have been two occasions when developers answered the impossible questions. Needless to say, offers were made to both of them.)

The one caveat of this technique is that it presupposes the person asking the question is fairly technical and can properly judge the answers. Many developers can spit crap with a straight face. I must confess there have been occasions when I have been in interviews where people would make up the most incredible answers to cut and and dry questions. I would hold my tongue and just listen because the answers were so outrageous they became entertaining. Of course, none of those people ever got jobs--but it was fun to listen.

I used to do approximately 6-8 of these almost back-to-back. This was a fabulous filter and worked great for me.

---

As Annakata pointed out, many of your better developers will not take a tests online. They often assume (and rightfully so) that many of the companies who use them are a) clueless when it comes to hiring and probably have a screwed up development environment and b) just screening as many candidates as they can--even though there may be little chance of getting the interview. By talking to people live, candidates know they are being seriously considered. Remember, the interview process works both ways. Not only are you filtering poor candidates. The better candidates are also filtering poor companies.

Answer 4

If you want to try rolling a test yourself try TestStack - they have free accounts and the underlying test engine seems pretty flexible. There are several other similar services but most of them are aimed squarely at the corporate market and are a lot more expensive.

Answer 5

We make up our own engineering test, with about 5-7 questions that takes about an hour. It's obvious if the candidate cheats (only 1 so far) and it does a good job of weeding out the no-hires.

Answer 6

Or you can just use CodeEval (www.codeeval.com)

The platform will allow you to ask all candidates to solve a programming challenge. Candidates have to send in their FULL LENGTH programs which are compiled/auto executed/scored at out end. The employer can then pick the ones with the highest score to call for a phone screen/on-site etc...

CodeEval can be used as a filter to weed out bad resumes and does this task 'intelligently'.

www.codeeval.com