I cannot find the MSI format specification. It says that its an open standard, but I don't see any documentation - just bits and pieces of information here and there.
I want to modify some .msi installer, but I want to understand first what it is doing.
Sounds like you got a misunderstanding of Microsoft's promise, MSI is not listed within.
In my work with MSI this link Windows Installer Tools & Tips was very helpfully.
Are you sure that .MSI database format is required to modify some installer? Have you heard about the Orca tool? If not, this MS KB article is for you:
The binary format is not open, I wouldn't even go as far as calling it a standard.
However I assume it's not the file format you're concerned with (which is undocumented), but rather the tables, actions, sequences, etc. that are visible when interfacing with the MSI via the Windows Installer SDK, API, Orca, etc.
Everything is covered in the SDK documentation, however for some reason search engines don't rank the documentation very highly. Also you can no longer download the Windows Installer SDK separately, it is now bundled as part of the Windows SDK.
MSI files are COM structured storage, which is described here - http://msdn.microsoft.com/en-us/library/aa380369%28VS.85%29.aspx