I recommend using something like ExtJs to build the GUI system in. I believe it's extensible enough that you should be able to implement whatever widgets are required to create your GUI builder.
As I see it, the main challenges are:
- Designing an effective interface that the designer can use. The best
way to learn how to do that is to look at all of the GUI designers out
there, and find ones that speak to you. I'm not a Windows guy, but VS
has a shockingly good GUI designer in it.
- Data binding, XML, and proper data design. There are several
metric tons of books on XML - personally I'd stay away from the
most complex bits, just make sure you have a reasonable schema and a
simple set of elements and attributes. Something like oXygen
greatly eases the pain of working in XML, and most of what you need to
know about schemas and such can be found on the excellent
w3schools.com
- Make sure to have the best
books on your target environment Not sure what the target language/
system for your design will be (what kinds of guis will people be building?)
You should probably look at source code for any open source GUI builder that you particularly like (Netbeans seems to have a rather good one). Might give you some inspiration.
You might also want to look into things like constraint satisfaction solvers, and learn a little bit of computational geometry. For the second, I cannot reccomend Geometric Tools for Computer Graphics highly enough. It's a very practical introduction and reference for the type of computational geometry you're likely to encounter.
Also, if you wanna save yourself some typing, you might have a look at CoffeeScript, which is more or less a "sweetened" version of JavaScript. Takes the edge off for me at least.