tags:

views:

418

answers:

6

I am trying to include huge string in my c++ programs, Its size is 20598617 characters , I am using #define to achieve it. I have a header file which contains this statement

#define "<huge string containing 20598617 characterd>"

When I try to compile the program I get error as fatal error C1060: compiler is out of heap space

I tried following command line options with no success

/Zm200
/Zm1000
/Zm2000

How can I make successful compilation of this program?

Platform: Windows 7

+8  A: 

Um, store the string in a separate resource of some sort and load it in? Seriously, in embedded land, you would have this as a separate resource and not hold it in RAM. On windows, I believe you can use .dlls or other external resources to handle this for you. Compilers aren't designed to hold this size of resources for you and they will fail.

Michael Dorgan
+16  A: 

You can't, not reliably. Even if it will compile, it's liable to break the runtime library, or the OS assumptions, and so forth.

If you tell us why you're trying to do it, we can offer lots of alternatives. Deciding how to handle indefinitely large data is a major part of programming.

Edited to add:

Rather than guess, I looked into MSDN:

Prior to adjacent strings being concatenated, a string cannot be longer than 16380 single-byte characters.

A Unicode string of about one half this length would also generate this error.

The page concludes:

You may want to store exceptionally large string literals (32K or more) in a custom resource or an external file.

What do other compilers say?

Further edited to add:

I created a file like this:

char s[] = {'x','x','x','x'};

I kept doubling the occurrences of 'x', testing each one as an #include file.

An 8388608 byte string succeeded; 16777216 bytes failed, with the "out of heap space" error.

egrunin
-1. The question has merit in and of itself. There may be viable alternatives, but it's also interesting to know the best way to directly incorporate large data into a program.
Eric
IMO "Why" is not important. I believe that when someone asks a precise question (like this one), it is better to give precise answer. When people want to do something weird, they (sometimes) have their reasons for that (if they don't, they'll learn something useful later).
SigTerm
+1. I think "incorporating large data into a program." is generally preferred "not done like this." The question has merit, but only as far as "What's the best workaround."
Precision
@Eric: My answer, though less useful than @Ira Baxter, was still correct. Downvote if the answer was factually incorrect, but I don't think I was.
egrunin
+1. For looking it up on MSDN and for putting it in a resource.
gbrandt
@SigTerm: re: "if they don't, they'll learn something useful later" If the OP posts some crazy thing they are trying to do, and it's painfully obvious to everyone that they are *probably* trying to do the wrong thing, then how are they going to learn something useful later if nobody questions their motivation, aside from using a resource other than SO?
John Dibling
@John Dibling: Dude, when you find correct solution yourself, you normally learn a lot of additional stuff (which you'll never find if you simply ask someone for help). Also... 1) this question doesn't qualify as "crazy". 2) "Probably" isn't enough. Unless you are 100% SURE that they ARE (notice: not "probably") doing wrong thing, you'll be wasting time telling OP things he/she didn't ask for (which is IMO quite rude). 3) Programmer should be using his/her own brain first. Asking other people is last resort.
SigTerm
@SigTerm: So, to paraphrase your stand: 1) When someone asks a technical question, it is rude to offer another approach to solve a problem, and 2) Programmers should strive to be islands, seeking no outside input from other programmers except in cases of last resort. Did I get that right? Dude?
John Dibling
@John Dibling: I'm not in the mood for strawman arguments. IMO: 1) When someone asks precise question, you should give precise answer to that question. You should not offer alternative, unless person asks for one. Answering question that wasn't asked is impolite and wastes time. Your time, and OPs time. 2) Programmer should know how to find answers himself, there for asking other people normally isn't necessary, and should be used when information cannot be obtained using other means, because it is slow. Also, asking for help too frequently may be a sign of laziness.
SigTerm
egrunin
... did you try char[] s= { 'xxxx', .... }?
Ira Baxter
@Ira: compiler didn't like that, truncated each to 'x'.
egrunin
@egrunin: Rats, sorry, I meant char[] s = { "xxxx" , ...};
Ira Baxter
@Ira: `char s[] = {"xx","xx"};` gives `error C2078: too many initializers`
egrunin
+3  A: 

Increase the compiler heap space.

amphetamachine
Hmm, this is actually a sensible answer (to a crazy question) and yet it's been downvoted. The last time I had a C program with a data segment of twelve megabytes, increasing the memory limits with `ulimit` worked.
Snake Plissken
@Snake - Thank you. I'm glad someone out there still expects something more from their OS and compiler.
amphetamachine
@amphetamachine Edit the post so I undownvote.
Precision
+8  A: 

Store the string to a file and just open and read it...

Its much cleaner/organized that way [i'm assuming that right now you have a file named blargh.h which contains that one #Define...]

ItzWarty
Could I ask why the one downvote o.o?
ItzWarty
+12  A: 

I suspect you are running into a design limit on the size of a character string. Most people really think that a million characters is long enough :-}

To avoid such design limits, I'd try not to put the whole thing into a single literal string. On the suspicion that #define macro bodies likewise have similar limits, I't try not to put the entire thing in a single #define, either.

Most C compilers will accept pretty big lists of individual characters as initializers. If you write

char c[]={ c1, c2, ...  c20598617 };

with the c_i being your individual characters, you may succeed. I've seen GCC2 applications where there were 2 million elements like this (apparantly they were loading some type of ROM image). You might even be able to group the c_i into blocks of K characters for K=100, 1000, 10000 as suits your tastes, and that might actually help the compiler.

You might also consider running your string through a compression algorithm, putting the compressed result into your C++ file by any of the above methods, and decompressing after the program was loaded. I suspect you can get a decompression algorithm into a few thousand bytes.

Ira Baxter
Qts resource system also used character arrays to store binary data last time i checked.
Georg Fritzsche
+1. If the string is reasonably compressible, it's probably faster to read the compressed form in and decompress it than to just read it in, considering the speed of the processor vs. the speed of the HDD.
Charles
@Charles: My last variant answer assumes he is building the compressed string *into* his load image. Why have a separate file to read, when the loader will do it for you, which is what I suspect the OP wanted?
Ira Baxter
@Ira Baxter, the data needs to be loaded whether it's done by the OS loader or the program itself. If compression means the program starts up a little bit faster, that would be a benefit.
Mark Ransom
Ah, yes, I was assuming that Charles was pushing reading a seperate file. Now I see we're all on the same page. I run Windows with a compressed file system for precisely this reason: CPUs are really fast, and disks are not.
Ira Baxter
+1  A: 

If your string comes from a large text or binary file, you may have luck with either the xxd -i command (to get everything in an array, per Ira Baxter's answer) or a variant of the bin2obj command (to get everything into a .o file you can link into the program).

Note that the string may not be null terminated in this case.

See answers to the earlier question, "How can I get the contents of a file at build time into my C++ string?"

(Also, as an aside: note the existence of the .xbm format.)

leander