Rendering Swing Components to an Offscreen buffer

Question

I have a Java (Swing) application, running on a 32-bit Windows 2008 Server, which needs to render it's output to an off-screen image (which is then picked up by another C++ application for rendering elsewhere). Most of the components render correctly, except in the odd case where a component which has just lost focus is occluded by another component, for example where there are two JComboBoxes close to each other, if the user interacts with the lower one, then clicks on the upper one so it's pull-down overlaps the other box.

In this situation, the component which has lost focus is rendered after the one occluding it, and so appears on top in the output. It renders correctly in the normal Java display (running full-screen on the primary display), and attempting to change the layers of the components in question does not help.

I am using a custom RepaintManager to paint the components to the offscreen image, and I assume the problem lies with the order in which addDirtyRegion() is called for each of the components in question, but I can't think of a good way of identifying when this particular state occurs in order to prevent it. Hacking it so that the object which has just lost focus is not repainted stops the problem, but obviously causes the bigger problem that it is not repainted in all other, normal, circumstances.

Is there any way of programmatically identifying this state, or of reordering things so that it does not occur?

Many thanks,

Nick

[edit] Added some code as an example:

Repaint manager and associated classes:

class NativeObject {
    private long nativeAddress = -1;

    protected void setNativeAddress(long address) {
        if ( nativeAddress != -1 ) {
            throw new IllegalStateException("native address already set for " + this);
        }
        this.nativeAddress = address;
        NativeObjectManager.getInstance().registerNativeObject(this, nativeAddress);
    }   
}

public class MemoryMappedFile extends NativeObject {
    private ByteBuffer buffer;

    public MemoryMappedFile(String name, int size)
    {
        setNativeAddress(create(name, size));
        buffer = getNativeBuffer();
    }

    private native long create(String name, int size);

    private native ByteBuffer getNativeBuffer();

    public native void lock();

    public native void unlock();

    public ByteBuffer getBuffer() {
        return buffer;
    }
}

private static class CustomRepaintManager extends RepaintManager{       
    class PaintLog {
        Rectangle bounds;
        Component component;
        Window window;

        PaintLog(int x, int y, int w, int h, Component c) {
            bounds = new Rectangle(x, y, w, h);
            this.component = c;
        }

        PaintLog(int x, int y, int w, int h, Window win) {
            bounds = new Rectangle(x, y, w, h);
            this.window= win;
        }
    }

    private MemoryMappedFile memoryMappedFile;
    private BufferedImage offscreenImage;
    private List<PaintLog> regions = new LinkedList<PaintLog>();
    private final Component contentPane;
    private Component lastFocusOwner;
    private Runnable sharedMemoryUpdater;
    private final IMetadataSource metadataSource;
    private Graphics2D offscreenGraphics;
    private Rectangle offscreenBounds = new Rectangle();
    private Rectangle repaintBounds = new Rectangle();

    public CustomRepaintManager(Component contentPane, IMetadataSource metadataSource) {
        this.contentPane = contentPane;
        this.metadataSource = metadataSource;
        offscreenBounds = new Rectangle(0, 0, 1920, 1080);  
        memoryMappedFile = new MemoryMappedFile("SystemConfigImage", offscreenBounds.width * offscreenBounds.height * 3 + 1024);
        offscreenImage = new BufferedImage(offscreenBounds.width, offscreenBounds.height, BufferedImage.TYPE_3BYTE_BGR);
        offscreenGraphics = offscreenImage.createGraphics();

        sharedMemoryUpdater = new Runnable(){
            @Override
            public void run()
            {
                updateSharedMemory();
            }
        };
    }

    private boolean getLocationRelativeToContentPane(Component c, Point screen) {
        if(!c.isVisible()) {
            return false;
        }

        if(c == contentPane) {
            return true;
        }

        Container parent = c.getParent();
        if(parent == null) {
            System.out.println("can't get parent!");
            return true;
        }

        if(!parent.isVisible()) {
            return false;
        }

        while ( !parent.equals(contentPane)) {
            screen.x += parent.getX();
            screen.y += parent.getY();
            parent = parent.getParent();

            if(parent == null) {
                System.out.println("can't get parent!");
                return true;
            }
            if(!parent.isVisible()) {
                return false;
            }
        }
        return true;
    }

    protected void updateSharedMemory() {
        if ( regions.isEmpty() ) return;

        List<PaintLog> regionsCopy = new LinkedList<PaintLog>();

        synchronized ( regions ) {
            regionsCopy.addAll(regions);
            regions.clear();
        }

        memoryMappedFile.lock();
        ByteBuffer mappedBuffer = memoryMappedFile.getBuffer();
        int imageDataSize = offscreenImage.getWidth() * offscreenImage.getHeight() * 3;
        mappedBuffer.position(imageDataSize);

        if ( mappedBuffer.getInt() == 0 ) {
            repaintBounds.setBounds(0, 0, 0, 0);
        } else {
            repaintBounds.x = mappedBuffer.getInt();
            repaintBounds.y = mappedBuffer.getInt();
            repaintBounds.width = mappedBuffer.getInt();
            repaintBounds.height = mappedBuffer.getInt();
        }

        for ( PaintLog region : regionsCopy ) {
            if ( region.component != null  && region.bounds.width > 0 && region.bounds.height > 0) {
                Point regionLocation = new Point(region.bounds.x, region.bounds.y);
                Point screenLocation = region.component.getLocation();
                boolean isVisible = getLocationRelativeToContentPane(region.component, screenLocation);

                if(!isVisible) {
                    continue;
                }

                if(region.bounds.x != 0 && screenLocation.x == 0 || region.bounds.y != 0 && screenLocation.y == 0){
                    region.bounds.width += region.bounds.x;
                    region.bounds.height += region.bounds.y;
                }

                Rectangle2D.intersect(region.bounds, offscreenBounds, region.bounds);

                if ( repaintBounds.isEmpty() ){
                    repaintBounds.setBounds( screenLocation.x, screenLocation.y, region.bounds.width, region.bounds.height);
                } else {
                    Rectangle2D.union(repaintBounds, new Rectangle(screenLocation.x, screenLocation.y, region.bounds.width, region.bounds.height), repaintBounds);
                }

                offscreenGraphics.translate(screenLocation.x, screenLocation.y);

                region.component.paint(offscreenGraphics);

                DataBufferByte byteBuffer = (DataBufferByte) offscreenImage.getData().getDataBuffer();
                int srcIndex = (screenLocation.x + screenLocation.y * offscreenImage.getWidth()) * 3;
                byte[] srcData = byteBuffer.getData();

                int maxY = Math.min(screenLocation.y + region.bounds.height, offscreenImage.getHeight());
                int regionLineSize = region.bounds.width * 3;

                for (int y = screenLocation.y; y < maxY; ++y){
                    mappedBuffer.position(srcIndex);

                    if ( srcIndex + regionLineSize > srcData.length ) {
                        break;
                    }
                    if ( srcIndex + regionLineSize > mappedBuffer.capacity() ) {
                        break;
                    }
                    try {
                        mappedBuffer.put( srcData, srcIndex, regionLineSize);
                    }
                    catch ( IndexOutOfBoundsException e) {
                        break;
                    }
                    srcIndex += 3 * offscreenImage.getWidth();
                }

                offscreenGraphics.translate(-screenLocation.x, -screenLocation.y);
                offscreenGraphics.setClip(null);

            } else if ( region.window != null ){    
                repaintBounds.setBounds(0, 0, offscreenImage.getWidth(), offscreenImage.getHeight() );

                offscreenGraphics.setClip(repaintBounds);

                contentPane.paint(offscreenGraphics);

                DataBufferByte byteBuffer = (DataBufferByte) offscreenImage.getData().getDataBuffer();
                mappedBuffer.position(0);
                mappedBuffer.put(byteBuffer.getData());
            }
        }

        mappedBuffer.position(imageDataSize);
        mappedBuffer.putInt(repaintBounds.isEmpty() ? 0 : 1);
        mappedBuffer.putInt(repaintBounds.x);
        mappedBuffer.putInt(repaintBounds.y);
        mappedBuffer.putInt(repaintBounds.width);
        mappedBuffer.putInt(repaintBounds.height);
        metadataSource.writeMetadata(mappedBuffer);

        memoryMappedFile.unlock();
    }

    @Override
    public void addDirtyRegion(JComponent c, int x, int y, int w, int h) {
        super.addDirtyRegion(c, x, y, w, h);
        synchronized ( regions ) {
            regions.add(new PaintLog(x, y, w, h, c));
        }
        SwingUtilities.invokeLater(sharedMemoryUpdater);
    }

    @Override
    public void addDirtyRegion(Window window, int x, int y, int w, int h) {
        super.addDirtyRegion(window, x, y, w, h);
        synchronized (regions) {    
            regions.add(new PaintLog(x, y, w, h, window));
        }
        SwingUtilities.invokeLater(sharedMemoryUpdater);
    }
}

The Panel that is having the problems:

private static class EncodingParametersPanel extends JPanel implements ActionListener
{
    private JLabel label1 = new JLabel();
    private JComboBox comboBox1 = new JComboBox();

    private JLabel label2 = new JLabel();
    private JComboBox comboBox2 = new JComboBox();

    private JLabel label3 = new JLabel();
    private JComboBox comboBox3 = new JComboBox();

    private JButton setButton = new JButton();

    public EncodingParametersPanel()
    {
        super(new BorderLayout());

        JPanel contentPanel = new JPanel(new VerticalFlowLayout());
        JPanel formatPanel = new JPanel(new VerticalFlowLayout());

        sdiFormatPanel.setBorder(BorderFactory.createTitledBorder(BorderFactory.createLoweredBevelBorder(), "Format"));

        label1.setText("First Option:");
        label2.setText("Second Option:");
        label3.setText("Third OPtion:");

        setButton.addActionListener(this);

        formatPanel.add(label1);
        formatPanel.add(comboBox1);
        formatPanel.add(label2);
        formatPanel.add(comboBox2);
        formatPanel.add(label3);
        formatPanel.add(comboBox3);

        contentPanel.add(formatPanel);

        contentPanel.add(setButton);

        add(contentPanel);
    }
}

Using this example, if the user interacts with comboBox2, then with comboBox1, the pull-down from comboBox1 overlaps comboBox2, but comboBox2 is redrawn on top of it.

Answer 1

I found a couple of things that could contribute to what you are seeing.

In the updateSharedMemory code for handling a repaint of a Window, the code calls contentPane.paint. This is the most likely culprit as the Window might not be your contentPane. The code for JPopupMenu (which is used by JComboBox) may choose to present the popup as a heavyweight component. So, the Window could be the popup of one of the JComboBoxes.

Additionally, the sharedMemoryUpdater is scheduled on the EDT where it will run once the event queue is empty. So, there may be a lag between when the addDirtyRegion is called and when updateSharedMemory is called. In updateSharedMemory there is a call to region.component.paint. If any of the already queued events change component, the actual results of the paint call could vary.

Some suggestions as a result of testing:

Create sharedMemoryUpdater like this:

    private Runnable scheduled = null;

    sharedMemoryUpdater = Runnable {
        public void run() {
            scheduled = null;
            updateSharedMemory();
        }
    }

Then, in addDirtyRegion

    if (scheduled == null) {
        scheduled = sharedMemoryUpdater;
        SwingUtilities.invokeLater(sharedMemoryUpdater);
    }

That will reduce the number of invocations of sharedMemoryUpdater (by 99% in my testing). As all of the calls to addDirtyRegion should be happening on the EDT, you shouldn't need a synchronize on scheduled, but adding won't hurt much.

Since there is a lag, the number of regions to process can become quite large. In my tests I saw it exceed 400 at one point.

These changes will cut the time spent manipulating the region list as it's faster to create 1 new list than create all the entries needed to copy the existing list.

private final Object regionLock = new Opject;
private List<PaintLog> regions = new LinkedList<PaintLog>();

// In addDirtyRegions()
synchronized(regionLock) {
    regions.add(...);
}

// In updateSharedMemory()
List<PaintLog> regionsCopy;
List<PaintLog> tmp = new LinkedList<PaintLog>()
synchronized(regionLock) {
    regionsCopy = regions;
    regions = tmp;
}