Startups and QA

views:

381

answers:

+5 Q:

Startups should have dedicated QA early in the process. Often, QA is added fairly late.

My two-part question is:

When should dedicated QA first be part of a startup effort and why?
What skills should the first QA members have (create and execute test scripts, test automation using common tools, write unit tests, plan and execute complex load and stability tests, etc.)?

+3 A:

As a sole developer for a start up let me provide my opinion:

When? As soon as you can afford it. I would prefer a dedicated tester right now as opposed to a second developer. I am horrible at testing my own apps. I subconciously know the exact method for accomplishing things, as such, I am always missing variations to how a real user uses the apps.
Skills? The more the better, I'd settle for a body to walkthrough a checklist manually running over the application. Anything helps

JoshBerke 2009-09-20 02:20:14

Most programmers I have worked with (including myself) aren't great at testing their own work. As the sole developer, do you plan time to create unit tests and some sort of formalized testing strategy? If so, how do you balance that against the demands to add functionality?

Eric J. 2009-09-20 02:48:36

At first I just winged, the app was small enough and it was new enough that I was preety good at running through it. After the first year I had to create a checklist of scenarios for me to work through. Today I have a part time intern who uses my checklist. As a startup our customers must come first we can't loose a single one...which makes it difficult. What we do is provide top knotch support. Some of the customers have my cell number. So features really comes first, we validate no major flaws, and then we respond very quickly.

JoshBerke 2009-09-20 03:16:21

While I would love to have what Bill suggests, and had I a QA team they would be involved in the design from the get-go. (In the other companies we worked we used to have QA sit in on all our initial design meetings). As a startup you need to be nimble, and make sure you don't miss an opportunity because your have to rigid a process.

JoshBerke 2009-09-20 03:21:32

@Josh: Yes, but you also have to have a team culture and that recognizes the importance of QA. To many people, QA are not respected because of prejudice; they're viewed as techies who aren't skilled enough to get coding jobs. That's the culture we have to change. If you have QA members who have both technical chops and the team's support, it can work.

Bill Karwin 2009-09-20 03:30:37

+12 A:

I'm going to offer an unconventional view:

The first QA members in a startup should have the skill of telling the development team when they haven't specified the project adequately (and helping them to reach this goal).

To me, Quality Assurance is not merely testing. Testing is Quality Control (QC). But you can't design tests for a product if you don't know what it's supposed to do. This is a situation that is all too common in a startup environment -- coders banging away furiously on a project without deciding what it is they're building.

The process of Quality Assurance begins before coding:

Define the requirements of the project specifically enough to test.
Implement the project using best practices, tools, methodology, etc.
Test to verify that what you built matches what you set out to build (this is QC).

That's why I say the first QA activity should be involved in the requirements specification phase.

Bill Karwin 2009-09-20 02:25:36

It's a sad statement that your view is, in fact, unconventional. That has been my mantra at both very large and very small companies but I have rarely seen it executed well.

Eric J. 2009-09-20 02:51:56

@Eric: My belief is that people want to solve technology problems using technology. But requirements definition requires human judgment, it can't be automated. It's hard to decide "this feature is in, that feature is out" but it may be the most important single thing we can do toward the goal of Quality.

Bill Karwin 2009-09-20 02:56:38

@Bill: Agree 100%. Not just in vs. out, but also "this feature is not defined clearly enough that all stakeholders share a common understanding of what it should do." That one issue causes untold man years of wasteful development.

Eric J. 2009-09-20 03:04:28

@Eric: The most common solution for this in a startup is that there is one person who has a vision, and the role of benevolent dictator. `:-)` If that person *also* has some self-discipline to resist "second system syndrome" (cf. The Mythical Man-Month), then it can all come together.

Bill Karwin 2009-09-20 03:25:10

@Bill: How do you find QA people who can execute this encompassing role well?

Eric J. 2009-09-20 03:43:16

Bill Karwin 2009-09-20 03:54:21

Another idea: find an older, experienced software engineer who knows how to work well with others and knows how to write good engineering specifications. They don't necessarily need to know the latest technology by heart, because requirements definition and writing are pretty much technology-independent skills (not to mention communicating). The "QA mentor" role might be a great opportunity to counter ageism in hiring.

Bill Karwin 2009-09-20 03:58:20

@Bill: Both are great suggestions, thanks.

Eric J. 2009-09-20 04:09:05

yes, but too many developers take on the QA tasks themselves thereby removing the separation between development and planning.

djangofan 2009-09-22 16:26:46

+1 A:

QA can be helpful, but until you have the funding to ensure that the product should get finished, adding a body just for testing is risky. If adding a new developer will help ensure the product is done before funding runs out then you may need to change priorities and get a new developer.

The developers may need to QA, but if that is the case, then the person that did the development for a feature should not be testing it.

You may want to have your QA person also be in charge of the continuous integration build, for example, so he may need to wear several hats, since everyone on the team will also be wearing several hats. Find someone that can be very flexible, so that they can help free up developers to do their development.

James Black 2009-09-20 02:26:18

That's an interesting idea to have the QA person also do CI. At least, if they're able to manage that process from a skills (or at least initiative/quick learner) perspective, it provides more options to handle peaks in demand on programmers vs. QA.

Eric J. 2009-09-20 02:51:03

For a startup, specialists are not very useful. Besides how much QA work is needed if you have one or two developers? Then you can have them talk with possible customers and get better requirements, and be more agile.

James Black 2009-09-20 03:59:31

As soon as your company is of sufficient size and profitability to allow specialization.

Most small firms require that folks be generalists and wear more than one hat.

No magic answer here, but the key thing for a small business is that every person has to bring in more revenue than they're paid; multiples are better than increments. That's a fact, no matter how much IT wants independent testers.

I'd imagine that small firms might have to be creative and do things like ask friends to test alpha versions for gift cards instead of money, or work with a trusted client.

duffymo 2009-09-20 02:30:18

A different model is that it's the owner who wears several hats, and hires specialists.

ChrisW 2009-09-20 02:44:44

There are generally different views about when the startup is "of sufficient size". Founders often want to keep headcount down to manage burn rate, while the IT team wants to have someone besides themselves test the code. Both are legitimate concerns. How would you strike the balance?

Eric J. 2009-09-20 02:45:11

+1 A:

As a QA engineer by trade, I'd say it's good to have dedicated QA as early as you can swing it. During the initial development cycle, QA will bring a different perspective to feature planning - how fast should this run? How high does it scale? How many users can one installation support simultaneously?

A good QA engineer is constantly finding new ways to break things. Having that focus from the start is always better than bringing in QA later on in the process, demoralizing the devs who thought they were doing pretty good until the new QA folks pile hundreds of bug reports on them, which then need to be triaged (is this even a bug? If QA had been involved from the start, they'd know that this is the intended behavior! etc) and then actually fixed.

And, let's be honest here... if we're talking a startup, they're not going to be writing test plans or producing useful automation. Test plans are awesome once you have a stable product and you have some assurances that the entire UI isn't going to change, and automation is great for catching regression, but during initial development, there won't be any regressions because you haven't shipped anything yet to regress from.

Really, the important skills a good QA engineer needs are communication, stubbornness, and the ability to come up to speed fast. If QA finds a bug but can't explain clearly and concisely how it happened, what they were doing, and how to reproduce it, their usefulness rapidly dwindles.

clee 2009-09-20 02:31:30

I appreciate your comments, especially about how the role must evolve over time due to the changing nature of the software (unstable feature set and rapidly changing early on, more stable as the product matures).

Eric J. 2009-09-20 02:46:13

+3 A:

Typically, QA isn't needed until after 1.0 and the product has a revenue stream.

It just needs to "work".

MathGladiator 2009-09-20 02:47:18

How do you know whether "it works"? My boss, who started by testing the code himself, used to say that "You get what you inspect."

ChrisW 2009-09-20 02:52:35

@ZenGeneral, I reluctantly agree (despite my other answer), because it's often true that a software project "done right" takes too long to reach market for the business requirements of a startup.

Bill Karwin 2009-09-20 03:22:23

@ChrisW "it works" when it makes money. As a software guy, this attitude sucks. As a business executive looking at the hungry faces of employees, it is the best course for a startup.

MathGladiator 2009-09-20 06:30:00

Ah yes. I was thinking of the old days, where you shipped to distributors, where pulling a release (because it was buggy) would be expensive (distributors returning your boxed product, and loss of face), and so you wanted to know **before you shipped it** whether it worked.

ChrisW 2009-09-20 11:48:15

+7 A:

I'm a strong believer in Agile and Lean Software Development. Please find here after some quotes from the fabulous Mary Poppendieck about defects:

"Inspection to find defects is WASTE. Inspection to prevent defects is essential."
"The job of Testing is not to find defects, the job of Testing is to prevent defects"
"A quality process builds quality into the code. If you routinely find defects during verification, your process is defective."

And a last one:

"If you have separate test and fix cycles, you're testing too late."

So, to answer your questions, my point of view is the following:

The job of QA is to create a test-driven environment that makes defects virtually impossible. Thus QA precedes development, it does not follow it. So, yes, start QA early, as early possible.
Ideally, embed tester(s) into the product development team. Have them work on testing during the iterations. Try to ship fully working and tested increment at each iteration.
To meet this goal, you'll need very skilled people. They need to master most of the testing areas and related tools: building test plans, writing (automated) acceptance tests, setting up test datas, eventually performing load testing, etc. Knowledge of BDD is a big plus as executable specifications are highly appreciated by development teams.

Pascal Thivent 2009-09-20 03:54:31

@Pascal: I like the way you embed the quality process in the development process. Your answer is conceptually similar to Bill's, but comes at the issue from a different angle.

Eric J. 2009-09-20 04:00:53

The idea is indeed to perform all tasks required to get potentially deliverable software (this is the "Definition of Done") during each iteration. The goal is to limit the number of work-in-process items which is a key to reduce cycle time (see Little's law and the Queuing theory) and become more responsive to feedback.

Pascal Thivent 2009-09-20 20:07:54

There are tons of resources available regarding QA, Testers, etc. and your question (a very good one) has no simple answer. There is no magic formula for when, how many, where and how early they are involved in the process, etc.

Probably the first thing you should figure out is what your actual needs are. Most people (unfortunately) equate QA to just testing. Testing is just one aspect of a QA team. Most people start with dedicated testers before they grow into having a "real" QA team.

Quality Assurance is more about preventative maintenance, while testing is more about fact finding.

Are you after testers or something more?

In regard to tester skills, the number one skill is being an expert in the domain of the product. Everything else is secondary.

Atoms 2009-09-21 01:04:33

I agree 100% that QA should focus on preventing (many) errors from happening. That's also a very good comment about having domain experience. I think that's often overlooked because there's often a mentality of hiring bodies to do testing rather than hiring experts to lead the quality effort.

Eric J. 2009-09-21 16:23:15

+1 A:

According to me the answer is as follows:

regarding the first question, the QA should be involved during requirement analysis. In this way the QA can sit with the programmer to discuss with the system engineer to make a thorough understanding of the requirements. In this process the QA will have a clear view of the requirement and hence wirte the test cases when the programmer goes for coding.

As a result the programmer and QA will finish their respective jobs at the same time.

regarding the second question, the QA should have a fairly good knowledge of the environment he have to work, methodologies of testing as well as basic knowledge of automation testing. If it is a startup project he will get time to work on automation and hence automate the regression and sanity test cases. It will thus save time and effort as well money for the management to deliver the product.

PJ 2009-09-22 10:43:04

Before you have QA you have a Product Manager with the vision and the organization and you have development. Before you hire your first QA person the Product Manager should put on the QA hat and be able to define the requirements and get things going with an initial test framework. Then, as the project gets out of infancy you can hire QA to begin taking project management tasks from the project manager. The important thing is to hire QA and product manatger that are skilled enough so that development doesnt need to put on the QA or design hats.

djangofan 2009-09-22 16:31:11

Most startups I've experienced working in rarely have a QA/Test person on board full time.

One option, as mentioned, is to get someone who can wear mulitiple hats.

Another option would be to hire them in as contractors or freelancers as and when needed.

There's also the idea of crowd sourced testing to consider or finding someone who can work remotely if that suits your business model.

Rosie Sherry 2009-11-08 22:58:18

ansaurus

tags:

views:

answers:

Startups and QA

related questions