I would like to extract out the labels, error messages, button text etc from out the java code. There is too much discrepancy right now and changing things is a pain.
There are too many of these Strings to be implemented as constants in a java file. Ideally I would like it be something like a properties file.
Should I be using the ResourceBundle? This application is strictly English and will be in English for its life.
It's completely OK to use ResourceBundle not just for internationalization.
In your case, using a ResourceBundle will ease the maintenance of your application, for correcting a typo or changing the texts of your GUI you won't need to open Java code and recompile the whole thing. If you see the easy maintenance of the GUI as a requirement then the use of a ResourceBundle is justified.
I would question how much a ResourceBundle would help you, in that you have to keep the keys as consistent as the labels. From experience on an internationalized application, those don't stay consistent any easier than text. It does offer some advantages in terms of externalizing the file (allowing someone to check over all text in one location), but can be a hot spot in terms of updates in source control.
ResourceBundle has the advantage of GUI tools understanding it at the expense of you having to be sure to override the language setting of the host operating system.
You might also consider a GUI tool which allows you to just externalize the labels and the GUI layout in general, such as this one.