How to act upon hitting "Enter" when on "Cancel" button in JFileChooser?

Answer 1

You have to add an ActionListener to the JFileChooser and react on the action with the command String set to JFileChooser.APPROVE_SELECTION.

Answer 2

The program doesn't terminate on my platform.

I see normal operation on Mac OS X 10.5, Ubuntu 10 and Windows 7 using (variously) Java 5 and 6. I replaced your exit() with println() to see the event:

System.out.println(rootDirChooser.getSelectedFile().getName() + e.paramString());

It may help to specify your platform and version; if possible, verify correct installation as well.

I'm not sure I understand your goal; but, as an alternative, consider overriding approveSelection():

private static class MyChooser extends JFileChooser {

    @Override
    public void approveSelection() {
        super.approveSelection();
        System.out.println(this.getSelectedFile().getName());
    }
}

Addendum:

The goal is to have the action of hitting the "Enter" key while on the "Cancel" button be identical to clicking on the "Cancel" button.

As discussed in Key Bindings, you can change the action associated with VK_ENTER.

KeyStroke enter = KeyStroke.getKeyStroke(KeyEvent.VK_ENTER, 0);
InputMap map = chooser.getInputMap(JFileChooser.WHEN_ANCESTOR_OF_FOCUSED_COMPONENT);
map.put(enter, "cancelSelection");

If you want the change to occur only while the "Cancel" button has focus, you'll need to do it in a Focus Listener.

Addendum:

I found a solution that uses KeyboadFocusManager, instead. What do you think?

I can see pros & cons each way, so I've outlined both below. Using KeyboadFocusManager finds all buttons, but offers no locale independent way to distinguish among them; the Focus Listener approach can only see the approve button, and it's UI specific. Still, you might combine the approaches for better results. A second opinion wouldn't be out of order.

Addendum:

I've updated the code below to eliminate the need to know the localized name of the "Cancel" button and use key bindings.

import java.awt.EventQueue;
import java.awt.KeyboardFocusManager;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import java.awt.event.FocusEvent;
import java.awt.event.FocusListener;
import java.awt.event.KeyEvent;
import java.beans.PropertyChangeEvent;
import java.beans.PropertyChangeListener;
import javax.swing.InputMap;
import javax.swing.JButton;
import javax.swing.JFileChooser;
import javax.swing.JFrame;
import javax.swing.KeyStroke;
import javax.swing.plaf.metal.MetalFileChooserUI;

public final class FileChooserKeys
    implements ActionListener, FocusListener, PropertyChangeListener {

    private final JFileChooser chooser = new JFileChooser();
    private final MyChooserUI myUI = new MyChooserUI(chooser);
    private final KeyStroke enterKey =
        KeyStroke.getKeyStroke(KeyEvent.VK_ENTER, 0);

    private void create() {
        final JFrame frame = new JFrame();
        frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);

        chooser.addActionListener(this);
        myUI.installUI(chooser);
        myUI.getApproveButton(chooser).addFocusListener(this);
        KeyboardFocusManager focusManager =
            KeyboardFocusManager.getCurrentKeyboardFocusManager();
        focusManager.addPropertyChangeListener(this);

        frame.add(chooser);
        frame.pack();
        frame.setVisible(true);
    }

    @Override
    public void actionPerformed(ActionEvent e) {
        System.out.println(e.paramString());
    }

    @Override
    public void focusGained(FocusEvent e) {
        System.out.println("ApproveButton gained focus.");
    }

    @Override
    public void focusLost(FocusEvent e) {
        System.out.println("ApproveButton lost focus.");
    }

    @Override
    public void propertyChange(PropertyChangeEvent e) {
        Object o = e.getNewValue();
        InputMap map = chooser.getInputMap(
            JFileChooser.WHEN_ANCESTOR_OF_FOCUSED_COMPONENT);
        if (o instanceof JButton) {
            if ("focusOwner".equals(e.getPropertyName())) {
                JButton b = (JButton) o;
                String s = b.getText();
                boolean inApproved = b == myUI.getApproveButton(chooser);
                if (!(s == null || "".equals(s) || inApproved)) {
                    map.put(enterKey, "cancelSelection");
                } else {
                    map.put(enterKey, "approveSelection");
                }
            }
        }
    }

    private static class MyChooserUI extends MetalFileChooserUI {

        public MyChooserUI(JFileChooser b) {
            super(b);
        }

        @Override
        protected JButton getApproveButton(JFileChooser fc) {
            return super.getApproveButton(fc);
        }
    }

    public static void main(final String[] args) {
        EventQueue.invokeLater(new Runnable() {

            @Override
            public void run() {
                new FileChooserKeys().create();
            }
        });
    }
}