I'm looking for guidelines/best practices for building software architecture of a document rendering software. For example, something on how to create a modular software to be able to parse/render various document formats.
Can you recommend any books/web sources?
You can start by looking at Calibre, an ebook management freeware which does display various ebook formats.
To setup a development environment (sources in Python in a Bazaar repo), read this page.
When it comes to rendering various format, ebooks formats are a good source.
The main projects are:
FBReader is interesting as it supports many other formats.
That being said, some more general initiatives can offer some global guidelines:
If you want to find out about basics of fonts (which is IMO the necessary base for any display/rendering software), then Fonts and Encodings by Yannis Haralambous is a very detailed and complete description.