Very good question and there's no absolute right answer IMO.
Questions to ask yourself:
1) What's the impact of a key becoming known
2) What is the trust level in the company
3) How important is it for engineers to be able to produce release builds
Ideas I have used over the years include:
Stored in source control repository but with restricted 'secure_group' access
Pros
- Key proliferation is reduced
- Access permissions are controlled by
scm admins
Cons
- Release build is restricted to those
with secure permissions
- Requires implicit trust of secure group members
Keys injected by build system
Standard build contains dummy key(s).
Release builds are generated by build server which replaces or injects production keys
Pros
- No bottleneck on engineers when
building code
- Key management is
restricted to build server + admins
Cons
- All data/systems must support dummy key
- Build server becomes bottleneck/mission critical component
Custom DRM package
Create your own key package i.e. RSA encrypted header with session generated symmetric key to encrypt key data. DRM approach also allows you to do stuff like set package expiry time or maximum number of uses
Pros
- Keys can be encapsulated
- Keys can
be safely distributed
- Audit trail as key package is generated per user
on demand with pub/private key pair
Cons
- A lot of custom code
- All build systems need to be re-engineered to read key package data
- Key Package needs lib/API to extract and so engineer can still read key data
There are other options such as secure encryption server or two-pass authentication web sites to retrieve key data.
In my experience there is no perfect solution though I'd be very interested in hearing suggestions or opinions from the community
Hope that helps