As I understand it, XML is a subset of SGML conceived to simplify it and encourage a more widespread usage.
I guess most useful features was brought into XML, but are there any features in SGML powerful enough to motivate using it instead of XML (and accepting the overhead in complexity)?
It is impossible to answer unless you have a specific application in mind. You can get better indexing and searching with SGML, but I haven't found a case where it is a better option than XML. I'm sure they are out there, I just haven't found them.
The winning factor is the rendering maturity on multiple platforms of XML. The same compatibility just doesn't exist for SGML. However, if you can control the environment, that may not be a factor. XML just seems to provide 90% of the functionality for much less complexity.
No. The only reason to use SGML is if you work in the publishing industry where SGML is already part of the production process. (There may be other industries but that's the only one where I have experience).
Another way of looking at this is, even though there are some nice features in SGML, it's not worth paying for the tools unless you can clearly show financial benefit by using SGML (see "publishing industry").
One advantage of SGML, if you are editing it manually, is that you can abbreviate tags in various ways. What exactly you can do depends on the SGML declaration for the DTD you are using. For example, in DocBook, you can write things like <literal>foo</>. In HTML, you can omit end tags, which is why you can write <p>foo ... <p>bar without closing the tags. Or you can omit tags altogether, which is why <body> is optional in HTML.
But XML was invented exactly because all these variants make parsing SGML pretty complicated. You will probably find more capable XML editors than SGML editors nowadays.
As far as processing, both forms are equivalent, because you can convert back and forth, except in some corner cases.