What is the correct way to get the process size on Solaris, HP-UX and AIX? Should we use top or ps -o vsz or something else?
Yes, you are right to lock at the VSZ.
"ps u" will give you the VSZ and RSS, which are the virtual memory size and resident set size. The RSS is how much physical memory has been allocated to the process, and the VSZ is the virtual memory size of the process. If you have several copies of a program running, a lot of the memory in the VSZ will be shared between those processes.
The exact definitions of "vsize," "rss," "rprvt," "rshrd," and other obscure-looking abbreviations vary from OS to OS. The manual pages for the "top" and "ps" commands will have some sort of description, but all such descriptions are simplified greatly (or are based on long-extinct kernel implementations). "Process size" as a concept is fiendishly difficult to pin down in the general case. Answers in specific instances depend heavily on the actual memory management implementation in the OS, and are rarely as satisfying as the tidy "process size" concept that exists in the minds of most users (and most developers).
For example, none of those numbers (nor, likely, any combination of them) can be used to tell you exactly how many such processes can run at once in a given amount of free memory. But really, your best bet is to come at it from that end: why do you want this number, and what will you use it for? Given that information, I think you'll get more useful answers.
On Solaris, you can get detailed information on a process's memory usage with the pmap command. In particular, pmap -x <pid> shows you how much of a process's memory is shared and how much is specifically used by that process. This is useful for working out the "marginal" memory usage of a process -- with this technique you can avoid double-counting shared libraries.