Why and when to hire a build engineer

Question

My company has grown to thirty-five coders, in five separate groups. I manage one of the groups. The bulk of the code is C#, but it must build and run on both .NET and Mono. We use C/C++ for drivers and real-time control, IronPython for scripting.

We have a simple automated build system that is triggered by check-ins; it does a big build of everything and gripes when compilation or unit tests fail. This is all fine and good, if primitive.

There is a strong desire among many of the engineers to hire a build engineer, to manage all the Makefiles and whatnot, to do releases to verification, to poke at the unit tests, etc. As headcount is tight, my boss (VP SW) is reluctant. I am trying to muster arguments and specifics for my cause, to create a bulletproof argument.

I recall at Apple, we had like 24 build engineers for 600 developers (25:1). At another company with 150 engineers, we had 6 build people (also 25:1). It seems like a decent metric.

If you would, please relate your opinions on the merits and and value-add of hiring a build engineer, along with any supporting evidence (e.g. what's it like where you are).

Thanks!

Answer 1

Disclaimer: I am presently employed as a build engineer.

I would suggest that once you have 3 or more individual groups working on multiple projects you ought to have one person responsible for your build systems and build scripts across all projects/groups. There are two reasons for this:

1) This person becomes a resource any developer can use to delegate work to or get assistance with when dealing with things encountered at compile time, thus relieving the developers of futzing with the build system/scripts so they can focus on their own applications.

2) Someone who can make sure the builds across all projects adhere to good standards and practices. Trust me, it is not fun when every single group defines different conventions for their source layout and build system/scripts and it becomes a huge pain for a developer in one group to look at the projects of, let alone transition to, another group if there is a large difference between how groups do things with their source and builds. This person essentially gives you governance across all groups.

Answer 2

I'd be curious what technology you're using both in language and in build software. Also, the complexity of your software is going to affect the need for a dedicated build engineer.

If you're writing c/c++/objective-c and using make, well, I've never had much luck with make, so good luck to you.

If you're writing in Java or .Net, that's a different story. In the Java community, Maven is the best build tool I've come across. And in the .Net world, it's hard to beat TFS+MSBuild with a well defined project layout. In my experience, once you get these tools setup and everything is well defined you really don't need anyone to watch them.

If you're using Python/Ruby/Perl, then...what build process? I'm kidding. Well, kinda.

For me, it's all about the process, really. Having everything well defined as to how to layout projects and build scripts makes everything much easier. If you have all that figured out or if you have someone who knows how to figure it out, then I don't think a build engineer is much of a benefit. However, if you need help figuring those things out, that's a different story.

Answer 3

In my experience, build engineering is the redheaded stepchild of software development. Very necessary once you get to a certain size, yet near-impossible to make the case to management for an entire headcount that isn't writing code.

My suggestion: see if you can get your boss to compromise. Unless you are experiencing huge headaches with build management then you may not really need one person 100% dedicated to build management. Hire a junior dev that really wants to get into fulltime coding and sic him/her on the build management. Great experience for the junior dev, 70%-80% dedicated resource to build management for you. Internships might also be an option if you have a university nearby.

Answer 4

If your present product developers have a backlog of enhancement requests and bug fixes, then every hour your present product developers devote to managing the build system is an hour's delay in clearing the backlog. A new-hire build engineer can usually learn your existing build system and its quirks in less calendar time than a new-hire product developer can learn your existing products sufficiently to be able to begin reducing your backlog. New-hires add a communication burden on existing team members while they are learning the system, see Mythical Man-Month, but it has been my experience that it takes much less time for a product developer to explain how s/he builds a component than to explain a component's 8,000 line legacy doIt() routine written by a committee of no-longer-present ancestors.

Context-shifts are harmful to productivity. If you have many versions of many products including various one-offs for special customers, a build specialist will reduce the interruption frequency for your product developers.

How much more productive are your most experienced developers over a new hire? Can they fix in three minutes a bug which would require a new hire a week? If so, you really want them doing that, not wrangling the build.

Answer 5

whaley said it all.

I just would like to add that in a small team, a release engineer could for example also do the installations (think nightly installation, one per customer, demo...) and keep the installation procedures and guide sane and up-to-date.

It's a lot more work than managers think it is,