This is a result of my search on the subject as I was curious about your question and this is where I would start myself. Sorry I am not a specialist on the subject but hope it can kick-start you!
Seems floorplan optimization is a matter of combinatorial optimization.
As a developer, you'll want to tackle the theory behind it and most likely some proven algorithms. You might then be interested by books such as:
It's a bit more difficult to get links on this subject, but if you're a member of IEEE Xplore, you might want to look at this paper and other similar ones.
Finally, on the floorplan wikipedia entry, you'll notice this on sliceable floorplans that might give you your best starting point:
Sliceable floorplans have been used in
a number of early EDA tools for a
number of reasons. Sliceable
floorplans may be conveniently
represented by binary trees which
correspond to the order of slicing.
What is more important, a number of
NP-hard problems with floorplans have
polynomial time algorithms when
restricted to sliceable floorplans
Good luck!