Why Does SetString Take Less Memory in Delphi (with Unicode)?

Question

This is Delphi 2009, so Unicode applies.

I had some code that was loading strings from a buffer into a StringList as follows:

      var Buffer: TBytes; RecStart, RecEnd: PChar; S: string;

      FileStream.Read(Buffer[0], Size);

      repeat
         ... find next record RecStart and RecEnd that point into the buffer;        

         SetString(S, RecStart, RecEnd - RecStart);
         MyStringList.Add(S);
      until end of buffer

But during some modifications, I changed my logic so that I ended up adding the identical records, but as a strings derived separately and not through SetString, i.e.

      var SRecord: string;

      repeat
        SRecord := '';
        repeat
          SRecord := SRecord + ... processed line from the buffer;
        until end of record in the buffer

        MyStringList.Add(SRecord);
      until end of buffer

What I noticed was the memory use of the StringList went up from 52 MB to about 70 MB. That was an increase of over 30%.

To get back to my lower memory usage, I found I had to use SetString to create the string variable to add to my StringList as follows:

      repeat
        SRecord := '';
        repeat
          SRecord := SRecord + ... processed line from the buffer;
        until end of record in the buffer

        SetString(S, PChar(SRecord), length(SRecord));
        MyStringList.Add(S);
      until end of buffer

Inspecting and comparing S and SRecord, they are in all cases exactly the same. But adding SRecord to MyStringList uses much more memory than adding S.

Does anyone know what's going on and why the SetString saves memory?

---

Followup. I didn't think it would, but I checked just to make sure.

Neither:

  SetLength(SRecord, length(SRecord));

nor

  Trim(SRecord);

releases the excess space. The SetString seems to be required to do so.

Answer 1

If you concatenate the string, the memory manager will allocate more memory because it assumes that you add more and more text to it and allocates additional space for future concatenations. This way the allocation size of the string is much larger than the used size (depending on the used memory manager). If you use SetString, the allocation size of the new string is almost the same as the used size. And when the SRecord string goes out of scope and its ref-count becomes zero, the memory occupied by SRecord is released. So you end up with the smallest needed allocation size for your string.

Answer 2

Try to install memory manager filter (Get/SetMemoryManager), which passes all calls to GetMem/FreeMem to default memory manager, but it also performs stats garhtering. You'll probably see that both variants are equal in memory consumption.

It's just memory fragmentation.