tags:

views:

1062

answers:

5

Hi,

I am using D6 Professional and need to create a text file in a specific format from lots of small strings already in memory. For performance reasons, I am considering using a TMemoryStream to collate the file data, and then write it out to disk in one go via a TFileStream.

But I have a half forgotten memory (probably from pre-D6 days) of reading somewhere that TMemoryStream is inefficient, especially after it hits its Capacity size. My Delphi (and Windows API) skill is not good enough to check the Classes.pas code for myself.

(OFFTOPIC) especially code like this: (line 5152 of Classes.pas):
NewCapacity := (NewCapacity + (MemoryDelta - 1)) and not (MemoryDelta - 1);
(/OFFTOPIC)

Adding to my worry is that the conclusion of a related question http://stackoverflow.com/questions/486843/using-memorystream-to-write-out-to-xml
was not to use TMemoryStream, but didn't say why - whether due to TMemoryStream itself, or because there is sufficient buffering in the TFileStream or the I/O device driver, or just the specifics of the code in question.

Thanks for any advice
Regards,
PhilW.

+8  A: 

A normal TFileStram also does buffering, and that is sufficient to optimize I/O. Putting a MemoryStream in front only adds overhead.

Henk Holterman
Henk, do you know for sure that TFileStream is buffered, please? Over time I've read in many places that TFileStream is **not** buferred, while Delphi's implementation of textfile is. My practice confirms that - readln is awfully fast, while reading from TFileStream isn't, esp. when reading many small buffers in a loop. There are several Delphi implementations of a buffered filestream (e.g. TGpBufferedStream by Primoz Gabrijelcic), which also suggtests to me native filestream isn't buffered.
moodforaday
No, the Delphi Wrapper TFileStream doesn't do any buffering but the Win32 stream does (albeit limited) buffering.
Henk Holterman
The "lets keep it simple" approach is always good. Thanks for the advice.
PhilW
+2  A: 

You can use a TStringList, and call SaveToFile method. TStringList has an advantage that memory are not copied when you add a string to it.

Another option is the Jedi JCL class TJclBufferedStream.

Marcelo Rocha
+5  A: 

The TFileStream itself doesn't perform the buffering, that is handled by the OS and is generally sufficient for most purposes.

My suggestion would be to build a method to write your data to a stream, and then pass a TSTream parameter to this method. This way you can test different options easily, without impacting your program.

For example:

Procedure TForm1.StreamMyObjects(aStream : tStream);
begin
  aStream.Write( MyString[1], Length( MyString ) * SizeOf( Char ));
  aStream.Write( CRLF, Length( CRLF ) * SizeOf( Char ));
  aStream.Write( MyOtherString[1], Length( MyOtherString ) * SizeOf( Char ));
  aStream.Write( CRLF, Length( CRLF ) * SizeOf( Char ));
end;

In the JCL, as before mentioned, there is a TJclBufferedStream which you can then test against to see if there is any performance benefit, which will vary based on what your writing, and how much your writing. For example the following will test a TFileStream, and a tJCLBufferedStream to see what the differences are (yes I know I'm missing the TRY/FINALLY):

var
  fstm : tFileSTream;
  fBufStm : tJCLBufferedStream;
  iTicks : Cardinal;
  fModes : word; // for SO formatting.
begin
  fModes := fmOpenReadWrite or fmCreate or fmShareExclusive;

  iTicks := GetTickCount;
  fstm := tFilestream.create('test1.txt',fModes);
  StreamMyObjects( fStm );
  fstm.free;
  ShowMessage('TEST1='+IntToSTr(GetTickCount-iTicks));

  iTicks := GetTickCount;
  fstm := tFilestream.create('test2.txt',fModes);
  fBufStm := tJclBufferedStream.create( fStm );
  StreamMyObjects( fBufStm );
  fBufStm.free;
  fstm.free;
  ShowMessage('TEST2='+IntToSTr(GetTickCount-iTicks));
end;

in my test, the following routine:

procedure TForm1.StreamMyObjects(aSTream: tStream);
var
  St : string;
  ix : integer;
begin
  for ix := 0 to 10000 do
    begin
      St := 'This is a string which is written to a stream. ' + IntToStr(ix);
      aStream.Write(st[1], Length(st) * SizeOf(Char) );
    end;
end;

returned 47 for the tFilestream, and 16 for the tJCLBufferedStream. Without the loop, the time is insignificant, which is why you need to test against your data...and how much your writing.

skamradt
A: 

I would investigate if it's worth the trouble to build a limited-size buffer of your own. It's slow to write little bits to a TFileStream (especially when delayed-write has been disabled on the storage volume), because the disk driver would have to locate the sector to write to on each little bit. So if you use a buffer that's the size of a few sectors (a safe guess would be 512KB or 1MB), and write the buffer when it's full, it would speed up your work, without having to worry what to do with a gargantuously bloating TMemoryStream.

Stijn Sanders
+2  A: 

As mentioned by the others adding a TMemoryStream in front of the TFileStream probably wont get you to where you need to get to.

Julian Bucknall produced, in a "The Delphi Magazine" article, a set of freeware units and classes to add functionality to TStream objects. I find them extremely useful in most of my dealings with TStream objects (especially TFileStream).

The link below is from google code search and provides access directly to the aaStrms.pas file. You will need the other units (aaIntDeq.pas, aaIntList.pas, aaRegEx.pas and aaStrBld.pas) to use the aaStrms.pas file. These are all non visual classes, so all you need to do is include the unit and instantiate the class.

http://www.google.com/codesearch/p?hl=en#7-hAM65i1Xc/disks/dmag70.zip|alfresco/AAStrms.pas&q=lang:pascal%20aaStrms

I would suggest that the TaaWriteBufferFilter class is the one you want to use.

HTH,

Ryan

Ryan J. Mills
+1 for helping me reduce my 30sec write/read of a 26MB binary file down to about 5secs! Cheers! :D
Ben Daniel