If you're stressing about X as a source of overhead on a modern Linux system you probably aren't looking in the right place. X was designed a really long time ago for computers much less powerful than a modern cell phone.
If you look at "top" and see X using memory, there's a lot of work to do to figure out the actual X overhead. There are memory maps that aren't "real" memory, and there are resources (such as big blocks of pixels) allocated on behalf of apps. Bottom line the memory shown for X in top isn't what one might think.
People also hear that X uses the "network" and think this is going to be a performance bottleneck. "Network" here means local UNIX domain socket, which has negligible overhead on modern Linux. Things that would bottleneck on the network, there are X extensions to make fast (shared memory pixmaps, DRI, etc.). Threads in-process wouldn't necessarily be faster than the X socket, because the bottlenecks have more to do with the inherent problem of coordinating multiple threads or processes accessing the same hardware, than with the minimal overhead of local sockets.
The multi-process setup has a lot of advantages, such as being much harder to crash. See Google Chrome for example, using multiple processes to be more robust - and it turns out, also to run fast. Less processes does not necessarily mean more modern.
There are many reasons apps using GTK don't transparently port to DirectFB. For Firefox, one is that it uses X directly sometimes. Also, some toolkit-independent stuff such as the browser plugin interface uses X directly. Flash plugin would not work on DirectFB for example. Even apps that don't use X directly would often assume the normal X-based desktop environment exists (GNOME, etc.).
Another issue with replacing X is driver support, where both of the better graphics cards (NVidia, ATI) have proprietary drivers that are a good bit more capable than the free drivers, and those proprietary drivers are tied to X.
And of course there's migration path. If you have hundreds of apps using X and no clear end-user downside to X, nobody is going to switch to something where no apps work. Most likely, the solution here would be a rootless X server running on a new window system, so old apps still work.
Old is not always bad. X was very well-designed by smart people, and that has allowed it to evolve and change and still work many years later.
Anyway all a long way of saying, basically switching away from X is tons of effort, it really works fine, and "works fine" has never applied to any of the alternatives (at least if you want to be able to run most apps on most hardware).
There are issues with X - such as the impossibility of doing an atomic screen update, something the Wayland project is looking at - but most of the issues are really cosmetic for users (e.g. non-atomic updates) or cosmetic for developers (old deprecated extensions and the like). It just isn't true that one could drop X and magically have something much smaller and faster. That's mostly based on people speculating that "old" and "uses network" must be slow and bloated, but again, X was designed for really really crappy hardware. I used to run X (and Emacs!) fine on my 386 with maybe 8 megs of RAM or something like that.