Well, the layout engine that powers IE is called Trident.
The rendering API used does depend on which version. I believe IE8+ uses DirectX 2D to do most of the rendering.
Previous versions would (I suppose, but not sure) use the standard GDI+ drawing functions for rendering.
Why don't you make a little application that uses the IE WebBrowser COM component, and then trace DLL calls made by the app?
Edit:
Easiest way to do a tracer application (and I can't believe I'm saying this) is to make a little VB6 app, add the Internet Explorer control onto the form, and have a button that triggers a single page navigate (use a really simple page, preferably a local one that you've made yourself). Run that app as an exe (not in the IDE), and then attach a profiler to it like API monitor, or just google 'win32 api trace'.