Whatever your pixel by pixel size, if it takes more than a few seconds for skilled users to complete the use of your window, then it shouldn’t be a dialog. Anything longer is annoying. You’d be asking your users to do too much work that is too easy to lose (e.g., by hitting Cancel accidentally) and too hard to re-entered (e.g., between sessions). If you have so many controls that the dialog needs to be 800 by anything, then it’s too many controls. 200,000 square pixels and 40 controls is the very most you should consider for a dialog. And tabs are nature’s way of saying your dialog is too complex.
Dialogs are for entering parameters to execute a single command on one or more data objects visible in the primary window for the dialog, which is why dialogs need to be small and simple. I suspect that’s not what you’re doing. Instead, you’re using a “dialog” to represent the main data objects and carry out a major task, not a single command.
What you want is a primary window, not a dialog, with all the support necessary for the complicated task you’ve set up for the user. That includes providing a means for users to save, retrieve, and copy their work. That means a menu bar and toolbar with all the standard commands, including help. The window should absolutely be modeless, and be resizable, maximizable, and minimizable.
Primary windows should be designed to work best at the size of most (over half) of your users’ screens. 1024x768 is generally fine for today’s laptop/desktop screens, not that you should use that much space if you don’t have to. If the user’s screen is smaller, or the user for any reason resizes your window below the design size, then scrollbars should appear to allow full access to all controls and content in the form –just like any primary window. The experience on the web indicates that scrolling is not a showstopper for forms.
Beyond that, tabs may be used in a primary window to increase the number of controls on it. You said you don’t want to hear about multiple windows, but multiple windows should be used instead of tabs if the user will be comparing data across tabs/windows. You can also fit more in a given primary window size by using a compact presentation (I describe this at http://www.zuschlogin.com/?p=42), but test such an approach on your users before committing to it.