I'm experimenting with Xlib to gain a deeper understanding of how Linux GUI programs work. I've written a simple program that display "Hello, world" in a window, but it looks rather anachronistic since the text is not anti-aliased.
What is the best way to display anti-aliased text in X11? How is anti-aliasing implemented in GTK, Qt, and other toolkits?
The X protocol's text-rendering facilities do not support anti-aliasing and aren't used much these days. (I think the reason is that the X font protocol doesn't have any place for an alpha channel.)
GTK and Qt render text in the client using the FreeType library, getting a pixmap with an alpha channel as the result. If the X server supports the RENDER extension, the client can send that pixmap to the server to have it blended onto the display using its alpha channel. If the X server doesn't support RENDER, the client has to retrieve the region of the screen where the text is to be displayed (taking a small screenshot, basically), do the alpha blending client-side, and send the resulting opaque pixmap back to the X server to be displayed.
FreeType is at the wrong level of the stack. It will only allow you to draw glyphs at certain places. Typically you need at least a font selection mechanism (supplied by Fontconfig) and a shaping engine (supplied by Pango or Qt).
Both Pango and Qt use a forked version of an abandoned FreeType layout engine, but this is being reconciled into the HarfBuzz project.
See also this post by Behdad Esfahbod: Pango vs HarfBuzz, and this longer and more comprehensive document: State of Text Rendering.