I don't exactly know how it usually works, so please tell me where to dig..
I need to create a tamper-resistant app installation file. Basically, if for some reason setup file(s) has been changed it suppose to tell about it and prevent any further installation.
How it works? I suppose it's something to do with md5 hash.
Is it possible to embed that stuff into a CustomActions installer module?
This is impossible. Sure, you can hash one part of the installation package. But then you need to have another part that checks the hash, and that can be modified. This has been tried repeatedly, and every time it has been broken due to this flaw.
You can easily detect inadvertent corruption, but that's not the same as being tamper-proof.
Most installation systems will handle this for you. They'll check the hash of the installer to ensure it hasnt been modified. Its not something you typically have to go out of your way to implement if you are using a decent installer system.
The complexity of the solution will depend on whether you want to detect whether you want to detect whether the file is different to the one that you built, or whether the application creates the file the first time it runs (so the application won't know in advance what the hash would be). If the first of these is what you want then one option would be to sign the installer, executable and the file (I think it's possible to sign a file other than a .exe or .dll). Or you could embed the file inside the .exe as a resource, along with its hash.
If I remember right the VS 2010 setup package creates an InstallShield installation package. What you want to do is digitally sign that package.
So what you need to do to meet your goals: 1) Figure out what the actual installer application is. It's likely InstallShield if you're using VS, there are alternatives to InstallShield available. 2) Go through the process of building a setup package with that tool 3) Read the directions from the install builder application to learn how to sign their packages
You will likely need to purchase verification from a security vendor like Verisign. Unverified signed files are not much better than signed files.
This is the most reliable way for your setup packages to self validate.
The next best alternative is to checksum the file (usually MD5) and recommend that people installing your software check that against the file they have before they run it.
What you're essentially asking for is Authenticode. It's an integrated part of Windows, and digitally signing your installer's EXE will allow you to ensure that it has not been tampered with between you and your user.
You should also note that when you digitally sign, UAC prompts will show your company name and a blue banner (instead of "Unknown" and an intimidating yellow banner).
However, a digitally signed EXE will not be prevented from running if the signature is invalid. To do that, your installer should validate its Authenticode signature.
If you only need to verify that the .exe file is properly distributed, calculate the md5 hash of the file after you yourself compile it, then have users compare their copy of the file's hash when they get it. If the hashes match, then the files should be uncorrupted.
To calculate a file's hash, check out this tutorial. I haven't personally used it, but it looks like it should be a good start for you.
Yeah.. I decided to put some executable on PreBuild event of the Setup Project, and that .exe will parse .vdproj file, get every single file name involved in installation and calculate its hash and save this information into an encoded file.
Then in OnInstall method of my CustomActions library, that file will be decoded and all hashes will be compared (at this time all files would be already unpacked into a temp folder). And if there is a problem setup will be stopped and all the temporary files will be deleted. Voilà and we have simple recipe. Of course there is always a possibility that CustomActions library could be reverse engineered, but I'm not gonna bring that to my boss, and I guess even if this matters I can use obfuscator to appease him.