tags:

views:

55

answers:

2

Before anyone says it, I know this isn't the way it should be done, but it's the way it was done and I'm trying to support it without rewriting it all.
I can assure you this isn't the worst bit by far.

The problem occurs when the application reads an entire file into a string variable. Normally this work OK because the files are small, but one user created a file of 107MB and that falls over.

intFreeFile = FreeFile
Open strFilename For Binary Access Read As intFreeFile
ReadFile = String(LOF(intFreeFile), " ")
Get intFreeFile, , ReadFile
Close intFreeFile

Now, it doesn't fall over at the line

ReadFile = String(LOF(intFreeFile), " ")

but on the

Get intFreeFile, , ReadFile

So what's going on here, surely the String has done the memory allocation so why would it complain about running out of memory on the Get?

A: 

You do not need that "GET" call, just remove it, you are already putting the file into a string, so there is no need to use the GET call.

ReadFile = Input(LOF(intFreeFile), intFreeFile)
kyndigs
The line ReadFile = String(LOF(intFreeFile), " ")just allocates a string of length LOF(intFreeFile) and fills it with spaces.We still need the GET to read the contents of the file into the string.
kenneedham
Misread it edited my answer
kyndigs
A: 

Usually reading a file involves some buffering, which takes space. I'm guessing here, but I'd look at the space needed for byte to character conversion. VB6 strings are 16 bit, but (binary) files are 8 bit. You'll need 107MB for the file content, plus 214 MB for the converted results. The string allocation only reserves the 214 MB.

MSalters
The file is a massive chunk of XML. Most of the space being XMLised images. I'll investigate what you say though.
kenneedham
@kenneedham: sounds like a case where it's reasonable to say "Sorry, we don't support that". Or add a special case that loads one byte at a time if the filesize is too large. (Actual disk I/O will still be buffered by the OS; you won't get 100 million physical I/Os)
MSalters
Indeed. Although to be fair, it's not really the users fault. If the application lets them do it then it should work. In this case it allows them to download the data but then not upload it again after they modify the spatial data. The whole thing needs re-writing but I'm sure we all know the answer I'd get if I suggest that.
kenneedham
Without seeing more code it would be hard to say how extensive a rewrite is required. Reading and processing in 64K chunks would go a long way toward improving scalability though. This is why "suck it all into RAM, the files are small" shouldn't even be done in throwaway programs: bad habits are formed that let you down in the crunch. We run into this with coders who try to use the XML DOM for everything. They have no clue about using the SAX parser. Usually we don't renew their contracts, let alone let them start working on server-side code.
Bob Riemersma
I have, in fact, changed it to process it in small chunks so the VB side of thing works now. However, the server side Java code (written by the same large, well known company) falls over with an Out of Memory exception now. I think it's time to tell the users not to try loading such large datasets.
kenneedham
@kenneedham: That's a bit unfriendly. It's often more useful to define a "large file support" project. Leave it to responsible managament to decide how much priority that project gets.
MSalters
I already know how much priority that project would get. Government cutbacks and all that. I can be much more diplomatic when actually speaking to end users.
kenneedham