Tips for hiring good testers?

Question

Hi all,

We're looking to add a few good testers (err "qa engineers") to our team.

It's been my experience in the past that the "10-1" rule for developers (good devs are 10 times more productive than mediocre devs) is even more prevalent for testers. Most testers I've worked with found a decent amount of issues, but there's a few stand-out testers that I've worked with that could rip applications apart and make the dev team cry.

So the question is -- how do you find these rockstar testers? The web is full of exams and tips to separate out the great from mediocre developers, but whats the magic interview questions and tests to find the amazing testers?

We're about to start the second round of interviewing with the people that have passed the phone screen (aka idiot test) and any suggestions you have for this challenge greatly appreciated.

Answer 1

Whip up a small obviously-demo application with a few obvious, less-obvious, and hidden issues. Let the candidate loose on it. Observe how they test it, maybe with a senior tester to help judge.

Answer 2

In addition to having them actually test things, have them write some bug reports.

I can tell you that as a software developer, the bane of my existence is QA people who don't provide any useful information in their bug reports. Even if someone can find every bug in the universe, they're useless if they can't tell me how they did it!

Finding the bugs is only part of the story. What's equally (if not more) important is the fact that they can effectively communicate the problems to the development team so that they can be reproduced and fixed.

A poorly written bug report can waste a lot of a developer's time just trying to figure out what the bug is before they can even begin to work on fixing it. It's these poorly written bug reports that usually make the dev teams cry rather than the act of finding bugs.

At a minimum, here's what I need as a developer to effectively do my job:

1) Software version (and build number) where the bug occurred
2) Description of the bug
3) Detailed list of steps to reproduce the bug

Bad QA people won't even provide #3 or will give you such vague information that it's not at all helpful and sometimes it's so bad that you don't even know what the hell they're talking about. You almost always wind up talking to a bad QA person several times before you can even start working on the bug.

Good QA people will provide adequate information so that the developer can reproduce the bug without having to even talk to the person who filed the bug report.

Great QA people will go the extra mile and try to really nail down the bug for you by trying alternate paths to the bug until they come to a very specific cause of it.

Completely contrived example to show my point

Let's say I have a print feature in an application. There are multiple ways to get to the print feature (File -> print, button on the toolbar, etc).

There's a crash that happens when printing more than 9 pages of a specific document foo.txt but it only crashes when using the Print icon on the toolbar.

Bad QA bug report: Application crashes when printing a document.

Good QA bug report: Application crashes when printing foo.txt from the Print icon on the toolbar (and foo.txt is attached to the bug report). Printing other documents does not cause the crash.

Great QA bug report: Application crashes when printing foo.txt from the Print icon on the toolbar (and foo.txt is attached to the bug report). Printing other documents does not cause the crash. Also found that printing from the File menu does not cause the crash and crash doesn't happen if I only print the first 9 pages of the document.

Answer 3

"whats the magic interview questions and tests to find the amazing testers?"

If you think there are magic questions, then you are already in trouble. Do you use magic questions for hiring developers, too?

If you aren't used to hiring testers, then find someone that is and have them help.

Answer 4

Not interview questions, but just a few things to watch for, with no particular order:

1. Testing instincts, you can find out about this using a demo application and see if the bugs are detected
2. Shell scripting skills, testers with great scripting skill can be xx times more efficient than the ones who aren't.
3. Communication skill, typical testers spend more time communicating the bugs and coordinating the fixes
4. Testing mind set, e.g. does he/she enjoy coming up with devious thoughts?

Answer 5

In building a test team, you will no doubt want to hire people with a mix of talents. You will need testers which have subject matter expertise in your project's problem space. You will also need testers who are technical - have the coding skills to create test automation suites and utilities to support manual testing. Sometimes you may be lucky enough to find a person who can fill both roles. Often though, these will be different individuals. The SME and technical testers each bring something different to the table.

Answer 6

One approach: give the potential tester an application (user interface, command line, whatever you usually work with) and a test case. Now, not just any test case, take a badly written, vague, maybe partially uncomprehensive test case. Tell the potential tester to roll up the sleeves. What I would expect is constructive criticism towards the test case. For a test report credibility the biggest holes in the test case should be filled with something (either offering to edit the test case, or describing the executed tests in a test report). The tester should also sport some creativity in steps that call for "try different variables". Your application should also have something worth bug reporting appearing in this test testing, just to see how the applicant approaches a possible bug.

In my opinion one of the big issues is the bug hunt in an interview. I don't think that many people are comfortable trying out different things and narrowing down the exact bug and workarounds for it when someone is watching over their shoulder. So the result is less likely to be excellent, even if you have a real star sitting on the other side of the table.

Answer 7

Lots of good advice already given here already; a couple more thoughts I'd add:

• get an understanding of the types of applications they've tested in the past and ask questions about the apps. A tester that really gets into the apps s/he is testing will probably do the same for you. If their experience is in testing desktop apps, and you're producing web apps, or vice-versa, think about the nuances that are unique (web apps: back button, browser differences, font settings, etc., etc.; desktop apps: installation processes, hardware configs, etc.) and try to see if they understand the differences and are ready to adapt if necessary.
• Another key area is test planning -- just as it's important to design code before writing it, any non-trival testing exercise should be planned out and documented. Give them information about the small app someone else mentioned, and ask them to outline a test plan. What questions do they ask? Do they ask who makes the final ship/launch decision and what their criteria are? Then, ask suggested by another, look at a bug report.
• Do they think of their test plan in modules or sections that can be reused as the system under test evolves over time?
• Experience with automation tools, and the judgement of when to use them, is also important.
• What tools have they used in the past to track bugs and communicate with programmers and/or management? Can they express how much of the testing job is complete, how much remains, and the risks to the plan (i.e., the time to test something written by a new programmer, or in a new technology is harder to predict than something the organization has more experience in)?

Good luck -- success here can be as critical as in hiring programmers.

Answer 8

Our QA Manager at the company I work at has put a lot of thought into this topic. He wrote an article about it that can be found here

Here is a quick summary of aspects he looks at when looking for software testers:

In my experience, there are attributes common to good software testers that indicate potential for almost any testing position, regardless of industry or development technology. For almost a decade, I have often used the following as a guideline of the basic attributes I look for:

• Customer-focussed, Quality-minded
• Troubleshooting and Problem-solving skills
• Good communicator – written and spoken
• Thinks like a Hacker – Analytical and likes to break software for fun or the challenge.

For a senior tester, I would add the following basic attributes to the above list:

• Critical Thinking and Systems Thinking skills
• Reverse Engineering and/or Modelling skills
• Adaptable and flexible
• Understands the financial cost of Quality
• Ethical, has integrity
• Commitment to learning, open-minded

Answer 9

It all depends on what kind of QA person you looking to hire- I have following suggestions based on who you looking to hire

1. To hire Automation expert - Give him to record some script and then evaluate the quality and the detail coverage of his script
2. To hire Sys/integration testing expert - Give him a use case and ask him to come up with the test scenarios related to it and check how detail and creative a person can get to cover most of the scenarios
3. To hire UAT testers - Just select a person who has the previous business experience working in the similar area

Answer 10

Find the ones that want to know how things work. They will be willing to let their minds turn and come up with creative ways to break your stuff. If they can answer questions pertaining to how things work they are probably fit for such a task.

Answer 11

Technical knowledge is a good to have for a QA person. He has to understand the working of an application from a much wider prospective than a developer, who would most of the time be restricted to his own component. The ability to develop small hacks and a perceived sense of knowing what can go wrong in a given component.

Another thing that a tester should have is the ability to ask questions. He should be curious to know what and why and also get to the how. He should not assume things, as once he does start assuming things, his ability as a tester goes down.

He should be able to write good and clear bug reports. The steps to reproduce a bug is a major bane for most developers ad we get a lot of questions after a bug has been written, which could have been easily resolved, if the bug report was comprehensive and written clearly. Hard to reproduce issues should have substantial evidence with them, so that these can be resolved without spending time on justifying a "cannot reproduce".

You have put in a good analogy of a tester with a product manager, these two roles are the ones who know the overall working of the product, while not actually getting into the internals. But a tester would be more keen on what ways can he/she devise to break the product, while a product manager, is more concerned with joining the pieces up and getting a perception of what the user actually would look for.

I would choose a person who is both technically sound and has a clear understanding of testing techniques. His logical and analytical abilities should be very good. Languages and Tools can be taught (if a basic knowledge of a HLL is there), but not logic and analysis of a problem/product.

My best has been to give the person being interviewed a use case like the address bar of the IE and let him find test cases or give him a function like memcopy and work out wat can go wrong with it and what all is to be tested to release the same.

For the above example, he should be able to nail down the minor issues quickly (you are the judge for this) and then get on to borderline cases and issues which are hard to find. He should be in control of the tests, not the tests be in control of him.

Answer 12

a) Case Reduction

As per 17 of 26. A tester needs to be able to reduce defect reproduction steps to the simplest failing case.

You interview for this by presenting the candidate with a contrived defect that appears in the course of a moderately complex use case in your actual product. If time is tight, provide an anecdotal description of that unabridged use case and ask how they would document it. A good candidate will begin with an attempt to get the failure to occur in fewer steps.

b) Intelligent Diagnosis

A concise failing case is a good start. Next, great testers provide relevant contextual information that helps the developer fix the bug ASAP.

Many bugs are trivial to fix once the developer knows where to look. A few minutes of diagnostic work by a competent tester can go a long way toward this.

E.g., if your team is building a web application, hire testers who are capable of using a Javascript debugger, know how to check web server logs, and can access the version control change set where the defect appeared, etc. In many cases the source of a defect will be transparent even to non-dev testers when they look at this info. Including the relevant bits in the defect report can save lots of time for the developer.

You interview for this by presenting the candidate with a contrived defect in your actual system and provide access to the resources above. Monitor the extent to which they use the diagnostic tools as they document the defect.

If time is tight, ask some basic screening questions relevant to your business and tools. E.g. given shell access to a directory with web server error logs, what command would you use to monitor log events?

c) Personality

You've worked with effective testers in the past, so you have some idea of the personality attributes you are looking for.

Use your own judgement to evaluate this. But remember to mind your own judgement. Weight it more heavily than technical competency when making your hire decision.

You can train for technical competency. Personality is another matter.

You interview for this by asking the candidate a series of open-ended questions and giving him or her a chance to speak at length on some subject that they should be familiar with. Ask about interactions with former co-workers, bosses, customers, etc. Who was your favorite / least favorite, etc.

Answer 13

Hey.
You have some great answers above: Bug reports, communication skills, waching candidate at work with some app. I really liked one about shortening the test case to hunt the bug, and one with criticizing badly written test case. Those two were less obvious yet so important.
But In addition I would add one more question, ask him or her to propose tests but not for the application. Ask for possible tests for a ball pen (or something else, commonly known yet not related to software). Very simple, yet checks if she or he can be creative, how they approach the problem. Can he or she show some constructive criticism in the way they think. Is candidate ready to jump on this level of abstraction? Does proposed test cases make sense?
Good tester should think of at least 20 tests without big problems (unless they can't handle stress of job interview). Additionally ask candidate to organize those tests in test suits. How he would group them? What tests should be performed more often than others? This way you will know if he can he bring some value regardless of technology he will need to face.