I am trying to understand how various ebook applications compile their output into a self-contained executable file with no dependencies when you run the exe. How can it be done? Do they use real embedded compiler?
They simply include a loader/viewer of the ebook that is paired with ebook data itself inside the executable.
There are many ways to do it, first of all you should understand if the binary really has no dependencies or if it searches for libraries (for example acrobat ones), then you can program a viewer that loads its data directly from the binary either in a vector-based view (smaller) either on a image-based view (like pdfs without text being ocr-ed)
I'm assuming that you're talking about a pdf-ebook.. for every other format the idea remains the same: pairing the ebook data with the viewer
The ebook application probably creates an application with an integrated web browser. The application will be pre-compiled and the only thing missing will be the html content. This will be added as a resource. A full compiler isn't necessary. All is needed is to add the resources to an existing executable.
See this article on how html pages can be added to a resource dll. Resource hacker is a utility that can extract/replace resources from any executable. No compiler is needed to do this.
The simplest approach, used by many archivers sfx modules, is to have precompiled stub, and then append created information to this stub (in your case compiled ebook). Then, when exe is executed is detect start of information block (for example with some predefined byte sequence) and extract information for play.
You can even embed external references in your exe, for example some tools, include .dll files as resources, which is extracted on run to temp folder and then loaded from there.
Of course there is more complex ways to embed information to existent executable, for example BeginUpdateResource - UpdateResource - EndUpdateResource sequence on WinNT is another way to embed your information into executable (as resource).
Ebook exe application maybe put your ebook in resource in .exe and try use a embedded webbrowser of pdf activex to load ebook.