What is the best process where code is checkedin by developers, installer is created by build engineer and release to QA to test the installer.
Should the installer be release to QA without unit testing by Dev. If dev do some changes then they should wait until QA to report bugs.Or if installer first given to dev for unit testing and once they signoff then only it should be release to QA?
Do Dev need to integrate with installer? That is, is it likely that a new release of the installer will make need changes by dev?
My guess is that there is integration between installer and code, and therefore integration test is needed. Seems reasonable that Dev should be involved.
So:
Determine where responsibility for Integration Test lies: if with Build Engineer, then release from them to QA. If with Dev, then first give to Dev and thence to QA.
Below is a link to a blog entry describing a new project to add unit testing to custom actions for the WiX installer. While this is very implementation specific, it may give you pointers on adding unit testing to your installer.
http://www.joyofsetup.com/2010/02/08/introducing-lux-declarative-unit-testing-for-custom-actions/
As a general guideline I would say how thoroughly you test your installer should be a function of how many people will be installing the product. If you are writing software for internal use at your company and there are e.g. 10 machines, then I would say just make sure the basic workings are tested. If your product will be released to the world and you anticipate hundreds of users, test every possible aspect of the process. Any installer bugs in this scenario will be a huge problem.
The times that I've done this, my approach has been to take the build engineer out of the process entirely. I automated building the installer as part of the same scripts that automate the build. The build engineer is responsible for automating builds, not writing and following checklists. (If you can write a checklist for the build process, then you can automate the build process. If you can't write a checklist for the build process, you have no business releasing software.)
Developers can build an entire release in their private development environment, and test any way they see fit (I recommend using a Virtual Machine with snapshots). Once the developer is happy, they commit their code to a SCM repository. A Continuous Integration engine (Hudson, CruiseControl, etc) monitors the repository and kicks off an integration build. Once the integration build is complete, the result is ready to send to QA for test. If you depend upon an installer that requires special licensed software to build, then instead of buying a licensed copy for each developer, just get a license for the CI server, and have the developers pick up installer builds from the CI server once its build complete.
If you want to have more process (shudder) then have the developers commit to a development branch, which is monitored by the CI server, which produces builds from the development branch. Only developers test the releases from the development branch. Once the development team is happy with the result, the changes from the development branch are promoted (merged) to the main branch, which is also monitored by the CI server, to produce a release for QA.
I've even worked in environments where there were (private) developer branches, which flow into team development branches, which flow into integration branches, which flow into release branches, each of which has a review board to determine when to promote changes from one branch to the next. That was insane, IMHO, but the QA staff loved it -- permanent employment for QA staff to monitor the process.