Fast read/write from file in delphi

Question

I am loading a file into a array in binary form this seems to take a while is there a better faster more efficent way to do this. i am using a similar method for writing back to the file.

procedure openfile(fname:string);
var
    myfile: file;
    filesizevalue,i:integer;
begin
  assignfile(myfile,fname);
  filesizevalue:=GetFileSize(fname); //my method
  SetLength(dataarray, filesizevalue);
  i:=0;
  Reset(myFile, 1);
  while not Eof(myFile) do
    begin
      BlockRead(myfile,dataarray[i], 1);
      i:=i+1;
    end;
  CloseFile(myfile);
end;

Answer 1

You generally shouldn't read files byte for byte. Use BlockRead with a larger value (512 or 1024 often are best) and use its return value to find out how many bytes were read.

If the size isn't too large (and your use of SetLength seems to support this), you can also use one BlockRead call reading the complete file at once. So, modifying your approach, this would be:

AssignFile(myfile,fname);
filesizevalue := GetFileSize(fname);
Reset(myFile, 1);
SetLength(dataarray, filesizevalue);
BlockRead(myFile, dataarray[0], filesizevalue);
CloseFile(myfile);

Perhaps you could also change the procedure to a boolean function named OpenAndReadFile and return false if the file couldn't be opened or read.

Answer 2

It depends on the file format. If it consists of several identical records, you can decide to create a file of that record type.

For example:

type
  TMyRecord = record
    fieldA: integer;

    ..
  end;
  TMyFile = file of TMyRecord;

  const
    cBufLen = 100 * sizeof(TMyRecord);
  var
    file: TMyFile;
    i : Integer;

  begin
    AssignFile(file, filename);
    Reset(file);
    i := 0;
    try
      while not Eof(file) do begin
        BlockRead(file, dataarray[i], cBufLen);
        Inc(i, cBufLen);
      end;
    finally
      CloseFile(file);
    end;
  end;

Answer 3

If it's a long enough file that reading it this way takes a noticeable amount of time, I'd use a stream instead. The block read will be a lot faster, and there's no loops to worry about. Something like this:

procedure openfile(fname:string);
var
    myfile: TFileStream;
    filesizevalue:integer;
begin
  filesizevalue:=GetFileSize(fname); //my method
  SetLength(dataarray, filesizevalue);
  myFile := TFileStream.Create(fname);
  try
    myFile.seek(0, soFromBeginning);
    myFile.ReadBuffer(dataarray[0], filesizevalue);
  finally
     myFile.free;
  end;
end;

It appears from your code that your record size is 1 byte long. If not, then change the read line to:

  myFile.ReadBuffer(dataarray[0], filesizevalue * SIZE);

or something similar.

Answer 4

If your really want to read a binary file fast, let windows worry about buffering ;-) by using Memory Mapped Files. Using this you can simple map a file to a memory location an read like it's an array.

Your function would become:

procedure openfile(fname:string);
var
    InputFile: TMappedFile;
begin
  InputFile := TMappedFile.Create;
  try
    InputFile.MapFile(fname);
    SetLength(dataarray, InputFile.Size);
    Move(PByteArray(InputFile.Content)[0], Result[0], InputFile.Size);
  finally
    InputFile.Free;
  end;
end;

But I would suggest not using the global variable dataarray, but either pass it as a var in the parameter, or use a function which returns the resulting array.

procedure ReadBytesFromFile(const AFileName : String; var ADestination : TByteArray);
var
    InputFile : TMappedFile;
begin
  InputFile := TMappedFile.Create;
  try
    InputFile.MapFile(AFileName);
    SetLength(ADestination, InputFile.Size);
    Move(PByteArray(InputFile.Content)[0], ADestination[0], InputFile.Size);
  finally
    InputFile.Free;
  end;
end;

The TMappedFile is from my article Fast reading of files using Memory Mapping, this article also contains an example of how to use it for more "advanced" binary files.

Answer 5

Look for a buffered TStream descendant. It will make your code a lot faster as the disk read is done fast, but you can loop through the buffer easily. There are various about, or you can write your own.

Answer 6

If you're feeling very bitheaded, you can bypass Win32 altogether and call the NT Native API function ZwOpenFile() which in my informal testing does shave a tiny bit off. Otherwise, I'd use Davy's Memory Mapped File solution above.