If your program uses a lot of windows resources it could be a Resource Heap shortage.
There is a registry entry that can be increased to raise the heap size for XP. For Vista Microsoft already sets the default value higher. I recommend changing the default 3072 to at least 8192.
This information is documented in the Knowledge Base Article ID 126962 (or search for "Out of Memory"). Additional details concerning the parameter values may be found in the Knowledge Base Article ID 184802.
I suggest you read the knowledgebase article but the basic info on the change is:
1) Run Registry Editor (REGEDT32.EXE).
2) From the HKEY_ LOCAL_MACHINE subtree, go to the following folder:
\System\CurrentControlSet\Control\Session Manager\SubSystem
3) On the right hand side of the screen double-click on the key:
windows
4) On the pop-up window you will see a very long field selected. Move the cursor near the beginning of the string looking for this (values may vary):
SharedSection=1024,3072,512
5) SharedSection specifies the System and desktop heaps using the following format: SharedSection=xxxx,yyyy,zzz where xxxx defines the maximum size of the system-wide heap (in kilobytes), yyyy defines the size of the per desktop heap, and zzz defines the size of the desktop heap for a "noninteractive" window station.
6) Change ONLY the yyyy value to 8192 (or larger) and press OK.
7) Exit the Registry Editor and reboot the PC for the change to take effect.
Good Luck