If you want to have your menu inside firefox window, then... I don't know the details, but the usual way in X is to make a X window (for example GtkWindow) with your GtkMenu. Then you should make firefox "swallow" that window, something like all the systray utilities do that (keywords:reparenting X window--should be enough). I guess there should be special XUL element for than, but I don't see any.
If you dont want to, you can simply make your own GtkWindow with GtkMenu and show it. I guess you should probably do this from another thread. Regarding your other question here -- it is fully possible to write an extension in C++, as long as you use Gecko SDK and Gecko API. You can then link with whatever library you want, including GTK.
I think you should ask this question on one of Mozilla's mailing lists, for example mozilla.dev.extensions (they are listed on http://www.mozilla.org/community/developer-forums.html).