Hi,
I got the answer for the question that i have asked.
Application Memory Allocators:
C and C++ developers must manually manage memory allocation and free memory. The default memory allocator is in the libc library.
Libc
Note that after free()is executed, the freed space is made available for further allocation by the application and not returned to the system. Memory is returned to the system only when the application terminates. That's why the application's process size usually never decreases. But for a long-running application, the application process size usually remains in a stable state because the freed memory can be reused. If this is not the case, then most likely the application is leaking memory, that is, allocated memory is used but never freed when no longer in use and the pointer to the allocated memory is not tracked by the application—basically lost.
The default memory allocator in libc is not good for multi-threaded applications when a concurrent malloc or free operation occurs frequently, especially for multi-threaded C++ applications. This is because creating and destroying C++ objects is part of C++ application development style. When the default libc allocator is used, the heap is protected by a single heap-lock, causing the default allocator not to be scalable for multi-threaded applications due to heavy lock contentions during malloc or free operations. It's easy to detect this problem with Solaris tools, as follows.
First, use prstat -mL -p to see if the application spends much time on locks; look at the LCK column. For example:
-bash-3.2# prstat -mL -p 14052
PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
14052 root 0.6 0.7 0.0 0.0 0.0 35 0.0 64 245 13 841 0 test_vector_/721
14052 root 1.0 0.0 0.0 0.0 0.0 35 0.0 64 287 5 731 0 test_vector_/941
14052 root 1.0 0.0 0.0 0.0 0.0 35 0.0 64 298 3 680 0 test_vector_/181
14052 root 1.0 0.1 0.0 0.0 0.0 35 0.0 64 298 3 1K 0 test_vector_/549
....
It shows that the application spend about 35 percent of its time waiting for locks.
Then, using the plockstat(1M) tool, find what locks the application is waiting for. For example, trace the application for 5 seconds with process ID 14052, and then filter the output with the c++filt utility for demangling C++ symbol names. (The c++filt utility is provided with the Sun Studio software.) Filtering through c++filt is not needed if the application is not a C++ application.
-bash-3.2# plockstat -e 5 -p 14052 | c++filt
Mutex block
Count nsec Lock Caller
9678 166540561 libc.so.1‘libc_malloc_lock libCrun.so.1‘void operator
delete(void*)+0x26
5530 197179848 libc.so.1‘libc_malloc_lock libCrun.so.1‘void*operator
new(unsigned)+0x38
......
From the preceding, you can see that the heap-lock libc_malloc_lock is heavily contended for and is a likely cause for the scaling issue. The solution for this scaling problem of the libc allocator is to use an improved memory allocator like the libumem library.
Also visit: http://developers.sun.com/solaris/articles/solaris_memory.html
Thanks for all who tried to answer my question,
Santhosh.