+3  A: 

A layout like that should not lead to an increasing memory footprint, or a slowdown.

Your description sounds more like a resource leak in your control (not Disposing graphic objects). Make sure that you know that the IDisposable interface implies the need of using(){} for Brush, Graphics and Image objects.

A short sample:

using (Graphics g = Graphics.FromImage(picture))
using (Brush fill = new SolidBrush(Color.Yellow))
{
    g.FillRectangle(fill, x0, y0, x1, y1);
}
Henk Holterman
Okey, I changed my approach and instead of adding controls to panel and stuff, I am drawing them directly as you suggested. Works find :) +25 for you. Tnx.
Anne