I am creating a .NET application (C#) that needs to use a lot of RAM. I recently knew that on 32-bit versions of Windows XP I can only use 2 GB, unless I use the /3Gb switch, and set the IMAGE_FILE_LARGE_ADDRESS_AWARE flag in the executable header. But since I'm developing a .NET application, I guess I cannot modify the executable directly, can I? So, what should I do to allow my application to utilize the 3 GB?
views:
1588answers:
7
+4
A:
/3GB switch is on the OS bootloader, not on your applications. (EDIT: It is also present in native C/C++ compilers, but not C# compiler) As far as your app is concerned, it will request memory and the OS will give it to your process. However you have access to 1 more gig (potentially, you don't always get 3gig depending on your hardware peripherals) before your program uses virtual memory.
As Marc Gavell has pointed out to me, you may need to run the command "editbin /LARGEADDRESSAWARE my.exe" as a post build option on your exe to enable this. Found a reference to an MS person speaking about it here: MS Forums
Might I suggest you look at your program and see whether you could rearchitect it to use less memory. Perhaps you could deal with a dataset in smaller chunks instead of trying to load the whole thing at once into memory?
Spence
2009-01-21 08:43:10
Thanks. The application is specificallly built to consume as much memory as needed in order to be as fast as possible (for number crunching and matrix operations). But do I get from your answer that the CLR has the IMAGE_FILE_LARGE_ADDRESS_AWARE bit set?
Hosam Aly
2009-01-21 08:47:14
@Hosam - you can always use `dumpbin /HEADERS` on your exe to find out; I've just checked, and it **doesn't** do this by default (i my test, at least)
Marc Gravell
2009-01-21 08:53:07
@Spence - no, you also need the exe-level header flag set.
Marc Gravell
2009-01-21 08:54:01
There is no compiler flag for it: http://msdn.microsoft.com/en-us/library/6ds95cz0(VS.80).aspxIn a mixed mode exe you definately would need it but are you sure for a c# exe?
Spence
2009-01-21 08:58:08
@Marc: Thanks for the tip. It's good to know about more tools. :)
Hosam Aly
2009-01-21 09:06:05
@Spence - well, my *understanding* is that it applies more to the process (via the OS) than the implementation. I could be wrong... I guess we could test by forcing heavy memory load etc...
Marc Gravell
2009-01-21 12:06:40
+5
A:
An .NET exe is still a standard PE file; so you could try using editbin /LARGEADDRESSAWARE to set the flag, but note that this won't work if you are using something like ClickOnce (since that maintains a cryptographic hash of the files).
However, note that you'll still have the same .NET limits in terms of the maximum size of a single object/array. For huge amounts of memory, x64 is a better idea.
Marc Gravell
2009-01-21 08:47:50
Thanks. I didn't know that it was a standard PE! So it just loads the JIT to compile the actual byte code stored in the executable? Do I understand correctly?
Hosam Aly
2009-01-21 08:49:57
(well, strictly it just asks the CLI to take over, and **that** does the JIT, but close enough ;-p)
Marc Gravell
2009-01-21 08:55:08
++ props for x64. Thats definately a good way to address the whole 3gb thing AND you don't have to recompile because of wonderful wonderful .net.
Spence
2009-01-21 09:01:28
Thanks Marc. :) I can't use 64-bit though, as we don't have 64-bit licenses (for Windows and 3rd-party applications). Hopefully we'll fix that soon.
Hosam Aly
2009-01-21 09:09:18
A:
As far as I remember, /3GB switch can only be used for Windows Server (2000 or 2003) but not for Windows XP. You need to write /3GB at the end of the boot.ini file. This way, for the applications, OS enables more memory which it normally allocates to use for kernel processes. However, /3GB does not mean you can use 3GB of memory for your application, it just can use more memory but it doesn't necessarily have to be 3GB. For .net applications, again as far as I remember, with /3GB switch you can use up to 1.8GB of memory. By the way, you might also want to check /PAE switch.
hakan
2009-01-21 08:53:57
Thanks. According to MSDN (http://msdn.microsoft.com/en-us/library/bb613473.aspx), the switch is supported on Windows XP Professional. As for the .NET limit, why would it be restricted to just 1.8? That's too little! (I am not talking about a contiguous array btw.)
Hosam Aly
2009-01-21 09:03:51
My knowledge mostly depends on the experiences that I have 3-4 years ago with .net 1.1 . I do not exactly know the main reason for that, even if I knew I don't remember now. However, whenever I got OOM exception, the dump file was showing 1.8GB total memory used, before /3GB switch it was 800MB.
hakan
2009-01-21 09:33:27
@haken - that sounds like it could have been related to either fragmentation of the LOH, or trying to allocate a single massive array (etc). Just thoughts.
Marc Gravell
2009-01-21 12:07:54
I thought, may be I was checking committed bytes instead of reserved bytes but it still doesn't explain the huge difference. We didn't have that big LO or sth like that. But it is better not to speculate on sth I don't remember exactly.
hakan
2009-01-21 12:57:04
+1
A:
Well, I'm not sure about this, but this is what I think:
A .NET executable can be compiled in two ways - platform specific and platform independant. By default they are platform independant, and the code is (as mentioned in other answers) JIT'ed to platform specific code when running the program.
Now, for example, if your executable is one of these platform-independant ones, and you run it on a 64-bit OS, it will be JIT'ed to 64-bit code, right? Thus it will be able to address way more that 3GB of RAM.
What I'm trying to say is - I don't think it matters at all what is written in the PE header. The actual amount of available RAM is determined by the .NET runtime, which in turn looks at the current platform and produces the best JIT'ed code it can.
I think you shouldn't worry about the /3GB switch as .NET will take care of it for you. Trust in the .NET! :)
Vilx-
2009-01-21 09:04:17
That's if I run it on a 64-bit OS, which I can't do currently. On a 32-bit OS, I need to set the large-address-aware flag.
Hosam Aly
2009-01-21 09:12:02
For classical executables, yes. But .NET executables aren't executed directly by Windows! They're first transformed by .NET runtime, which in result produces something completely different! It's not your .exe that runs, it's the one that .NET produces.
Vilx-
2009-01-21 09:26:36
So if the flag would have to be set anywhere, it would have to be set on the .exe that .NET produces, not your .exe!
Vilx-
2009-01-21 09:27:11
Your .net exe is actually a real exe with a PE header as Marc Gravell pointed out. This code loads, calls Cor_BindToRuntimeEx() which loads the CLR. The Clr then jumps to the first point of .net code in the EXE and begins execution. Therfore windows runs your exe just like the C++ exe :(.
Spence
2009-01-21 09:50:47
A:
You could try using Remoting via Named Pipes and get more memory by physically having more processes.
If you are doing any form of interop (and normal .Net sockets count here) you should create an object cache (e.g. with sockets a byte[] buffer) that allocates a large amount of these objects at application startup.
You should read this article.
Jonathan C Dickinson
2009-01-21 09:41:09
Thanks, but remoting would incur a very large overhead to my application.
Hosam Aly
2009-01-21 09:53:03
You might be surprised. You can get pretty good performance by using Named Pipes (they are the IPC channels). +Take note of the edit.
Jonathan C Dickinson
2009-01-22 06:08:24
+1
A:
You should also increase your process' maximum working set size: see the SetProcessWorkingSetSize API.
ChrisW
2009-01-21 14:06:58
Thank you. It is the first time I know about this function. Is it a must to call it? Or is it just-in-case, so that no other application takes my memory? (Do I need it if the machine is dedicated to my application?)
Hosam Aly
2009-01-21 14:32:22
If you don't call it then, when you exceed your maximum working set, the O/S will start to 'trim' your working set by paging your memory to the page file (which, swapping memory to and from disk, affects the performance of your program and of the O/S). So you *should* call it if you want to ...
ChrisW
2009-01-21 14:50:16
... actually be *using* all that *memory*. ALternatively if you're happy with using 3GB of *virtual* memory (not of real memory, of actual RAM) then you don't need to call it.
ChrisW
2009-01-21 14:51:30
A:
Is "editbin /LARGEADDRESSAWARE my.exe" working for C# applications? I was told that it works only for C++ applications. Could you please tell me if we need to update boot.ini file as well?
Thanks!
rk1962
2010-06-20 20:25:19