Lua runs out of memory

Question

I've written a complicated lua script which uses the lua sockets library. It reads a list of files from disk, sorts them by date and sends them to a HTTP process. The number of files on disk is around 65K.The memory usage in taskmanager doesn't exceed 200Mb.

After quite a while the script returns:

lua: not enough memory

I print out the current GC count at points and it never goes above 110Mb

local freeMem = collectgarbage('count');
print("GC Count : " .. freeMem/1024 .. " MB");

This is on a 32 bit windows machine.

What's the best way to diagnose this?

Answer 1

All memory goes through the single lua_Alloc function. This takes the form of:

typedef void* (*lua_Alloc) (void* ud, void* ptr, size_t oszie, size_t nsize);

All allocations, reallocations and frees go through this. The documentation for this can be found at this web page. You can easily write your own to track all memory operations. For example,

void* MyAlloc (void* ud, void* ptr, size_t osize, size_t nsize)
{
    (void)ud; (void)osize;   // Not used
    if (nsize == 0)
    {
        free(ptr)
        TrackSubtract(osize);
        return NULL;
    }
    else
    {
        void* p = realloc(ptr,nsize);
        TrackSubtract(osize);
        if (p) TrackAdd(nsize);
        return p;
    }
}

You can write the TrackAdd() and TrackSubtract() functions to whatever you want: output to a log; adjust a counter and so on.

To use your new function you pass a pointer to it when you create the Lua state:

lua_State* L = lua_newstate(&MyAlloc,0);

The documentation to lua_newstate is found here.

Good luck.

Answer 2

Use perfmon to monitor your process and add counters for private bytes and virtual bytes.

When your script ends with 'not enough memory' see the value of each counter. If you see sudden peaks in your memory usage, try to add more points in which you print the memory usage.