Help me improve my continuous deployment workflow

views:

464

answers:

+13 Q:

Help me improve my continuous deployment workflow

I've been developing a workflow for practicing a mostly automated continuous deployment cycle for a PHP project. I'd like some feedback on possible process or technical bottlenecks in this workflow, suggestions for improvement, and ideas for how to better automate and increase the ease-of-use for my team.

Core components:

Hudson CI server
Git and GitHub
PHPUnit unit tests
Selenium RC
Sauce OnDemand for automated, cross-browser, cloud testing with Selenium RC
Puppet for automating test server deployments
Gerrit for Git code review
Gerrit Trigger for Hudson

EDIT: I've changed the workflow graphic to take ircmaxwell's contributions into account by: removing PHPUnit's extension for Selenium RC and running those tests only as part of the QC stage; adding a QC stage; moving UI testing after code review but before merges; moving merges after the QC stage; moving deployment after the merge.

This workflow graphic describes the process. My questions / thoughts / concerns follow.

Continuous Deployment Workflow

My concerns / thoughts / questions:

Overall difficulty using this system.
Time involvement.
Difficulty employing Gerrit.
Difficulty employing Puppet.
We'll be deploying on Amazon EC2 instances later. If we're going about setting up Debian packages with Puppet and deploying to Linode slices now, is there a potential for a working deployment on Linode to break on EC2? Should we instead be doing our builds and deployments on EC2 from the get-go?
Another question re: EC2 and Puppet. We're also considering using Scalr as a solution. Would it make as much sense to avoid the overhead of Puppet for this alone and invest in Scalr instead? I have a secondary (ha!) concern here about cost; the Selenium tests shouldn't be running that often that EC2 build instances will be running 24/7, but for something like a five-minute build, paying for an hour of EC2 usage seems a bit much.
Possible process bottlenecks on merges.
Could "A" be moved?

Credits: Portions of this workflow are inspired by Digg's awesome post on continuous deployment. The workflow graphic above is inspired by the Android OS Project.

+3 A:

How many people are working on it? If you only have maybe 10 or 20 developers, I'm not sure it will make sense to put such an elaborate workflow into place. If you're managing 500, sure...

My personal feeling is KISS. Keep It Simple, Stupid... You want a process that's both efficient, and more important: simple. If it's complicated, either nobody is going to do it right, or after time parts will slip. If you make it simple, it will become second nature and after a few weeks nobody would question the process (Well, the semantics of it anyway)...

And the other personal feeling is always run all of your UNIT tests. That way, you can skip a whole decision tree in your flow chart. After all, what's more expensive, a few minutes of CPU time, or the brain cycles to understand the difference between the partial test passing and the massive test failing. Remember, a fail is a fail, and there's no practical reason that code should ever be shown to a reviewer that has the potential to fail the build.

Now, Selenium tests are typically quite expensive, so I might agree to push those off until after the reviewer approves. But you'll need to think about that one...

Oh, and if I was implementing this, I would put a formal QC stage in there. I want human testers to look at any changes that are being made. Yes, Selenium can verify the things you know about, but only a human can find things you didn't think of. Feed back their findings into new Selenium and Integration tests to prevent regressions...

ircmaxell 2010-09-13 19:06:53

We're a very small team at the moment, so I certainly agree that KISS is the way to go. That said, I'd like a process that scales, though that doesn't necessarily mean one that's complex. At the moment our process is continuous integration only and it has become basically second nature. Write and run tests on local; commit to master when ready; Hudson builds and runs tests on the server and reports back. Rinse and repeat.

Josh Smith 2010-09-13 19:10:26

As for the unit tests, also agree wholeheartedly. It's Selenium that'll be the slowest. So when I wrote in the workflow that it's a portion of unit tests, I really meant ALL the PHPUnit tests and none of the Selenium tests.

Josh Smith 2010-09-13 19:12:07

Well, remember that if you create your process right from the begining, you can scale it up in sections. So basically you'd have a series of stages (test, verify, qc and deploy, for example). Then, when you need to add process, you add it where it needs it. If you notice the verification stage is taking too much time, or isn't working right, you adjust that then. You don't need to have an elaborate system for it to scale. You just need an intelligently designed simple system that people believe in. If others aren't committed, it won't happen, no matter how good the workflow...

ircmaxell 2010-09-13 19:13:53

+1 for the edit re: QC. Though what I'm thinking is that in an alpha stage of the product, we can pushing code to production and getting feedback from testers, which can then be written into the new Selenium tests. Your thoughts on where/how to do it?

Josh Smith 2010-09-13 19:14:35

Well, IMHO you want a QC from pre-alpha all the way through. Otherwise you may miss an architectural bug, and it's far more expensive to fix later. Have your QC team write and maintain the Selenium tests. That way, it's in their interest to write them (since it makes their lives easier), and it's their problem to verify if one fails (that it wasn't a false positive). I'm a fan of "escalation". Where you have Dev->Review->QC->Production. One level can push to the next, but no further. So that way once it's out of one hand, they don't need to worry unless they hear back (and can continue on).

ircmaxell 2010-09-13 19:18:10

Gotcha re: QC and agreed. As per your overall sense that this is overly complex, I'm just wondering what steps you'd omit or simplify. Since Gerrit's all automated and all about code review, I'm not sure those aspects are really all that complex. My thought is that maybe I should eliminate using `PHPUnit`'s Selenium RC extension and instead write those tests for the `Sauce OnDemand` portion. Then after code review, Gerrit performs a merge rather than a build. That eliminates basically the "6th" and "7th" grid rows from the workflow.

Josh Smith 2010-09-13 19:29:12

Actually, now that I'm looking at it, I can't really see much to get rid of. Perhaps run the Sauce tests before the merge as well (since that's testing the code), and also run all unit tests when you are currently running the Sauce tests (since you're testing the merge, rather than just the local copy). I guess I'm just hesitant at any significant level of complexity (even automated) since it's going to be tougher to maintain or modify the automation later... Use automation to simply, but don't make it complex since then you're tied to the complexity (and when something breaks, have fun)...

ircmaxell 2010-09-13 19:36:27

I think what's nice about the automation is that `Hudson` and `Puppet` only play minor roles in the automation process. We don't have to change our build much to continue using `Hudson`. And `Gerrit` makes up the rest of the automation and is part-and-parcel with our code review.

Josh Smith 2010-09-13 19:51:49

Edited workflow graphic thanks to your contributions. Let me know what you think.

Josh Smith 2010-09-13 20:22:15

That's pretty close to what I would do (the focus on "stages", the dedicated QC steps, etc)... Looks good to me...

ircmaxell 2010-09-13 20:41:21

Thanks so much for your help! Really very deeply appreciate it.

Josh Smith 2010-09-13 20:47:12

+2 A:

Importent to make your tests extremely fast, i.e. no IO and ability to run parallel and distributed tests. Don't know how applicable it is with php, but if you can test units of code with in memory db and mock the environment you'll be better off.

If you have a QA/QC or any human in the way between the commit and production you would have a problem getting to a full continuous deployment. The key is your trust your testing, monitoring and auto response (immune system) enough to eliminate error prone process evolving humans from your system.

Eishay Smith 2010-09-14 04:03:39

+1 A:

I don't know whether this is relevant to PHP, but you can replace at least at least some of the code review stage with static analysis.

The quality of code reviews is relying on the quality of the reviewers, while static analysis relies on best practices and patterns, and is fully automatic. I'm not saying that code reviews should be abandoned, I simply think it can be done off-line.

See

http://en.wikipedia.org/wiki/Static_code_analysis

http://en.wikipedia.org/wiki/List_of_tools_for_static_code_analysis

Eran Harel 2010-09-14 07:05:10

Would you mind expanding upon this some? I'm honestly quite new to all of this (been developing for just under four months now) so I'd appreciate some further explication.

Josh Smith 2010-09-14 07:35:52

@Josh I edited my answer with some relevant links

Eran Harel 2010-09-14 11:47:40

@Eran Thanks so much for doing so. Much appreciated.

Josh Smith 2010-09-14 18:38:14

I'm quite ignorant about continuous deployment, but your workflow ending with "deploy code to production" sounds like having a continuous integrationnworkflow ending with "compile merged code", where is the feedback loop??

How to get that feedback? Maybe Eishay could be so kind to enlighten us on this topic.

erich 2010-09-17 17:00:14

ansaurus

tags:

views:

answers:

Help me improve my continuous deployment workflow

Core components:

My concerns / thoughts / questions:

related questions