Setting up a deployment / build / CI cycle for PHP projects

Question

I am a lone developer most of my time, working on a number of big, mainly PHP-based projects. I want to professionalize and automate how changes to the code base are handled, and create a Continuous Integration process that makes the transition to work in a team possible without having to make fundamental changes.

What I am doing right now is, I have a local test environment for every project; I use SVN for each project; changes are tested locally, and then transferred to the on-line version, usually via FTP. API documentation is generated manually from the source code; Unit tests are something I am getting into slowly, and it's not yet part of my daily routine.

The "build cycle" I am envisioning would do the following:

• A changeset gets checked into SVN after having been tested locally.

• I start the build process. The SVN HEAD revision gets checked out, modified if necessary, and made ready for upload.

• API Documentation gets generated automatically - if I haven't set it up in detail yet, using a default template, scanning the whole code base.

• The new revision is deployed to the remote location via FTP (Including some directory renaming, chmodding, importing databases, and the likes.) This is something I already like phing for very much, but I'm open for alternatives of course.

• Unit tests residing in a predefined location are run. I am informed about their failure or success using E-Mail, RSS or (preferably) HTML output that I can grab and put into a web page.

• (optionally) a end-user "changelog" text file in a pre-defined location gets updated with a pre-defined part of the commit message ("It is now possible to filter for both "foo" and "bar" at the same time). This message is not necessarily identical with the SVN commit message, which probably contains much more internal information.

• Stuff like code metrics, code style checking and so on are not my primary focus right now, but on the long run, they certainly will. Solutions that bring this out-of-the-box are very kindly looked upon.

I am looking for

• Feedback and experiences from people who are or were in a similar situation, and have successfully implemented a solution for this

• Especially, good step-by-step tutorials and walkthroughs on how to set this up

• Solutions that provide as much automation as possible, for example by creating a skeleton API, test cases and so on for each new project.

and also

• Product recommendations. What I know so far is phing/ant for building, and phpUnderControl or Hudson for the reporting part. I like them all as far as I can see, but I have of course no detailed experience with them.

I am swamped with work, so I have a strong inclination towards simple solutions. On the other hand, if a feature is missing, I'll cry about it being too limited. :) Point-and-click solutions are welcome, too. I am also to commercial product recommendations that can work with PHP projects.

My setup

I am working on Windows locally (7, to be exact) and most client projects are run on a LAMP stack, often on shared hosting (= no remote SSH). I am looking for solutions that I can run in my own environment. I am ready to set up a Linux VM for this, no problem. Hosted solutions are interesting for me only if they provide all of the aspects described, or are flexible enough to interact with the other parts of the process.

Bounty I am accepting the answer that I feel will give me the most mileage. There is a lot of excellent input here, I wish I could accept more than one answer. Thanks everyone!

Answer 1

I've recently begun the same kind of process, and am using Beanstalk for svn hosting.

There are two nifty features in the paid accounts (start at $15pm i think):

• deployment allows the user to create ftp targets for staging and production servers, which can be deployed at the click of a button (inc specifying a revision and branch)
• webhooks allow the user to set up a url that is called on each commit/deploy, passing across things like revision number, description and user. This could be used to update docs, run unit tests and update changelogs.

I'm sure there are other hosted or self-hosting svn servers with these two features, but beanstalk is the one i have experience of and it's working very, very well

There's also an API, which I imagine could be used to integrate deployment further in to your process.

Answer 2

The term you are looking for is: continous integration.

Here is an example of someone who uses GIT + phpundercontrol: http://maff.ailoo.net/2009/09/continuous-integration-phpundercontrol-git/

CruiseControl (which is a CI server), can use Hosted SVN/GIT as source. So you can even use it with GitHub or Beanstalk or something else.

Then you can integrate that with following kind of software:

• PHPUnit
• php-codesniffer
• phpdocumentor
• PHP Gcov
• PHPXref
• Yasca
• etc.

Well you could try this hosted CI: http://www.php-ci.net/hosting/create-project

Keep in mind though, that those tools need custom support, if you integrate them yourself.

Did you also thought about project management and patch management?

You can use Redmine for project management. It has integrated continuous integration support, but only as client side (not as CI server).

Try using a hosted SVN/GIT/etc. solution, because they will cover your backups and keep their servers running, so you can focus on development.

For a tut on how to setup Hudson, see: http://toptopic.wordpress.com/2009/02/26/php-and-hudson/

Answer 3

I use Atlassian's Bamboo continous integration server for my main PHP project (along with their other products such as fisheye (repository browsing), jira (issue tracker) and clover (code coverage)).

It supports SVN and now supports Git and it has a great user interface. It is available for linux, windows and mac and can run standalone on its own tomcat server which is great for people (like me) who does not like to take days to setup their tools). Although it may look expensive, being a lone developer myself I purchased the starter kit license for 10$ (10$ by software). This is great for small teams and it is worth the look.

Answer 4

I've been through buildbot, CruiseControl.net, CruiseControl and Hudson. All though I really liked CruiseControl*, it was just too much of a hassle with really complex dependency cases. buildbot is not easy to set up, but it's got a nice aura (I just like python, that's all). But hudson won over the former three because:

1. It's just easy to set up
2. It's easy to customize
3. It looks good and got nice overview functionality
4. It got point-and-click updates, for itself and all installed plugins. This is a really nice feature, that I appreciate more and more

Caveat: I only ever used linux as base for the above mentioned build servers (CC.net ran on mono), but they should all - according to the docs - run cross-platform.

Setting up a hudson server

Prerequisites:

• Java (1.5 will serve you just fine)
• Read access to the subversion server (I have a separate account for the hudson user)

From here, it's just:

java -jar hudson.war

This will run a small server instance right off your console, and you should be able to browse the installation at your localhost:8080, if you don't have anything else running on that port in advance (you can specify another port by passing the --httpPort=ANOTHER_HTTP_PORT option to the above command) and everything went well in the 'installation' process.

If you go to the available plugins directory, you'll find plugins for supporting your above mentioned tasks (subversion support is installed per default).

If that has whet you appetite, you should install a java application server, such as tomcat or jetty. Installation instructions are available for all major application servers

Update: Kohsuke Kawaguchi has constructed a windows service installer for hudson

Setting up a project in hudson

The links in the following walk-through assumes a running instance of hudson located at http://localhost:8080

1. Select new Job from the menu on the left
2. Give the job a name and tick Build a free-style software project on the list
3. Pressing 'ok' will take you to the configuration page of the job. All the options have a little question mark besides them. Pressing this will bring up a help text regarding the option.
4. Under the option group 'Source Code Management' you would be using Subversion. Hudson accepts both url access as well as local module access
5. Under the option group 'Build Triggers', you would use 'Poll SCM'. The syntax used here is that of cron, so polling the subversion repository every 5 minutes would be 5 * * * *
6. The process of building the project is specified under the option group 'Build'. If you already have an ant build file with all the targets you need, you're in luck. Just choose 'Invoke ant' and write the name of the target. The option group supports maven and shell commands as well out of the box, but there are also a plugin available for phing.
7. Tick off additional build actions in 'Post Build Actions', such as e-mail notifications or archiving of build artifacts.

For setting up processes for which hudson have no plugins, you can either call them directly through a shell script from within the build setup, or you could write you own plugin

Pitfalls:

• If you have it produce build artifacts, remember to have hudson clean up after itself in regular intervals.
• If you have more than 20 projects set up, consider not displaying their build status as the default main page on hudson

Good luck!

Answer 5

I am also a lone developer, though I do not have the same environment as you.

I do not use many of the products, or even types of products that you use, but I will give you my experience.

I run a TEST environment in parrallel with my PROD environment. I have no local testing per se. If it is too hard to get soemthing up into a real TEST environment, then I fix my build process. I don't see the point in testing locally, as the environments are different. UPDATE: The only thing I do locally is run "php -l" before I upload anything. Stops the stupid mistakes.

The build process works with whatever is in the current workspace, which includes uncommitted code. This is not everyone's cup of tea, but I am going to TEST very often. Everything gets committed before going to PROD.

Part of my build process (similar to yours) creates two META files. One contains the last (typically) 100 changes and also gives me the current changelist number. The shows me what changes are installed. The other contains the CLIENTSPEC (in Perforce terms) which shows me exactly what branches were used in this build. Together these give me reproducible builds.

I do not build straight to the target environment, but to a staging area on the server. I use SSH so this makes sense. This gives me a few advantages. Most importantly it avoids dying half way through a large upload. It also gives me a place to store META files, and all the build files are automatically archived (so I can go straight back to any build). The script also logs the update (so there is an entry in the log stream and I can see pre- and post-) and kicks all daemons (I use daemontools so "svc -t"). All of these are better off on the target machine.

One other issue is DB changes. I keep a master script of the DB schema, which I update every time the schema changes. Each of the changes also go into a changes.sql script, which is uploaded with the build to the staging area. The script is run as part of the install script.

Answer 6

Consider fazend.com, a free hosted CI platform, which automates configuration and installation procedures. You don't need to setup version control, bug tracking, CI server, test environment, etc. Everything is done on-demand.