I am currently making a Windows port for Mozilla XUL. My approach does not involve linear programming techniques like Steve mentioned, but it is a more object oriented approach. It is based on the Composite and Decorator design patterns.
The composite pattern allows you to create controls that have child controls, which in turn can have their own children. A control is responsible for positioning its child controls within its designated client rectangle.
For example suppose you want to implement a layout that positions its child controls horizontally. The layout algorithm then needs to calculate the width of each child control in order calculate the x offsets for each child control.
Calculating the width of a container is done by returning the sum of the widths of the child controls.
The Decorator classes can be used to add extra properties. For example a MarginDecorator can add spacing between child controls, a ScrollDecorator can scrollbars, etc...
It's a fun thing to do, I wish you good luck!