It looks like you're using a GridLayout, or perhaps a FlowLayout, neither being what you want. You probably want to use a BoxLayout, which respects its components preferred sizes.
final JPanel p = new JPanel();
p.setLayout(new BoxLayout(p, BoxLayout.X_AXIS));
p.add(leftPanel);
p.add(mainPanel);
p.add(rightPanel);
also- that is the preferred size. if you don't want to allow resizing, you can also set the maximum sizes as well.
if you're able, you may want to check out the MIG layout, but BoxLayout is also easy to use and in the java toolkit already.
Don't do this.
The whole point of layout managers is to allow dynamic resizing, which is necessary not just for user-resizable windows but also changing texts (internationalization) and different default font sizes and fonts.
If you just use the layout managers correctly, they will take care of panel sizes. To avoid having your components stretched out all over the screen when the user increases the window size, have an outermost panel with a left-aligned FlowLayout and the rest of the UI as its single child - that will give the UI its preferred size and any surplus is filled with the background color.