Originally, it meant:
An ellipsis (...) after a menu item means that after the item is chosen, the user will be asked for more information before the operation is carried out. Usually, the user must fill in a dialog box and click and OK button or its equivalent. Don't use the ellipsis when the dialog box that will appear is merely a confirmation or warning (for example, 'Save changes before quitting?').
(Apple Human Interface Guidelines, page 69)
Note that it did not mean "show a dialog box", even though that was often the consequence of this. For example, on Mac OS (not X), the "Options" button in the Page Setup window had no ellipsis, even though it showed a modal dialog box. No ellipsis is used because showing the options window is the operation.
(Tog on Interface, pages 46-47)
Of course, these days nobody cares about such things as human interface guidelines, not even Apple, so you can pretty much do what you want and still be more consistent than most any other application out there.