Suppose you develop for a customer that requires the utmost assurances of the provenance and process-compliance of the software you deliver to them. What measures can a development organization take to provide high-integrity software?
This was originally inspired by a couple questions about security practices for development systems over at ServerFault.
Everything I mention here has a range of possibilities, whose suitability is up to your budget and the degree of assurance you and your customers desire. Many of the steps below involve applications of cryptographic hashes and signatures, and that's not an accident. Applied with the proper technical controls, they represent an unambiguous mark that whoever (or whatever machine) possessed the signing key affirmed that the process was being followed properly.
Let's start at the delivery end, and work our way back. The basic measure here is to sign your binaries. Anyone who cares to should be able to verify the authenticity of that signature, and the meaning of that particular signature (this is a supported release, or this was generated by us, or anything else is possible).That means the public keys should be widely distributed, separate from the delivered binary. That could be by pre-arrangement with your customers, or that could be through a third-party signed certificate. In either case, if you care, verifying the signature should be a routine part of the acceptance process that your customers carry out. For the really paranoid (cough Apple cough), the hardware could do that signature check.
Take a step back in the binary's past now: by what process was the binary generated? Was it compiled and signed on a secured build server? Did a random developer click 'Build' in the IDE's toolbar? For non-production builds, it may be acceptable to use binaries generated by random developers. The key for signing release builds should probably only be available to the automated build process. Then, you can focus on ensuring the security of the one or few blessed build machines. As an aside, having designated build systems is also an opportunity to enforce automated tests and ensure the use of a validated tool chain (compiler versions and such). It may make sense to go all-out in protecting these machines, with very limited network access, strict authorization and auditing on logins, intrusion detection, etc.
Going further backward, what code is being built? All of the major open source projects release GPG-signed tarballs of the released source archive. Those using the popular DVCS like Git and Mercurial support similarly signed tags in the repository itself. For the build system to sign a binary as a release, it should have checked that it was building properly signed source of the stated version, and that the signature is from someone authorized to make a release.
Where does the code come from? Well, people write it and then they commit it to a repository. Before any code gets integrated into a release branch, you'll probably want to take measures to verify both who wrote that code, and that it's up to your quality standards.
The "who wrote it" (or realistically (unless you're watching over their shoulders), "who will vouch for it") part is straightforward: enforce strong authentication and audit logging on any write access to the repository. This can be passwords, SSH keys, GPG signatures (again, signed tags), OTP cryptotokens, etc. There's a whole industry built around providing authentication.
The "up to quality standards" can require somewhat more effort. If you have tests that must pass, the integration process needs to check that. Continuous integration tools can be really helpful here. If code must pass inspection (aka review), integration should be a direct result of a positive inspection report. To see an example implementation for this sort of policy, have a look at Gerrit.
As for how the code gets written, that's up to the good sense of your developers. Plenty of people have written at length about development process, tools, and techniques.