Direct file access in delphi

Question

my application opens files does transformations and saves the data out to another file..or possible the same file.. the file size changes but i dont know how big or small its gona be untill i see the data inside the first file..

At the moment i load the file into a dynamic array do all that i need to do in there then save it back... this was looking good untill i got to my testing stage where i found transforming multi gigabyte files on a system with 128mb ram caused some issues...LOL here is my code..

procedure openfile(fname:string);
var
  myfile: file;
  filesizevalue:integer;
begin
  AssignFile(myfile,fname);
  filesizevalue := GetFileSize(fname);
  Reset(myFile, 1);
  SetLength(dataarray, filesizevalue);
  BlockRead(myFile, dataarray[0], filesizevalue);
  CloseFile(myfile);
end;

what i need is direct file access to minimise ram usage.. thats what i think i need/ is this what i need can it be done in delphi

Answer 1

I don't think you'll get a "more direct" file access. Do you use all the data in the file? Otherwise you could perhaps use a stream and load only the data needed into memory. But if you use all the data, there's only one solution IMHO: read the file in chunks. But that highly depends on the kind of transformation you want to apply. If the transformation is not local (so that combined data elements are all in the same chunk), you gonna have problems.

Answer 2

You can also map parts of file directly into memory. That's definitely the most direct way. See What is the fastest way to Parse a line in Delphi for an example.

Answer 3

I'd look at using a TFileStream, perhaps one with buffering, but you need to show what you are doing with the data really because it is hard to determine the best strategy. As gabr says, one option is memory mapped files, the code of which is in his link but since it is my code, I'll add it here too!

procedure TMyReader.InitialiseMapping(szFilename : string);
var
//  nError : DWORD;
    bGood : boolean;
begin
    bGood := False;
    m_hFile := CreateFile(PChar(szFilename), GENERIC_READ, 0, nil, OPEN_EXISTING, 0, 0);
    if m_hFile <> INVALID_HANDLE_VALUE then
    begin
        m_hMap := CreateFileMapping(m_hFile, nil, PAGE_READONLY, 0, 0, nil);
        if m_hMap <> 0 then
        begin
            m_pMemory := MapViewOfFile(m_hMap, FILE_MAP_READ, 0, 0, 0);
            if m_pMemory <> nil then
            begin
                    htlArray := Pointer(Integer(m_pMemory) + m_dwDataPosition);
                    bGood := True;
            end
            else
            begin
 //                      nError := GetLastError;
            end;
       end;
    end;
    if not bGood then
        raise Exception.Create('Unable to map token file into memory');
end;

Answer 4

If problem permits, you can use BlockRead and BlockWrite to read chunk of input file, process it and then write that chunk to output file. Something like this:

  AssignFile(inFile,inFname); 
  AssignFile(outFile,outFname); 
  repeat      
    BlockRead(inFile, buff, SizeOf(buff), bytesRead);
    ProcessBuffer(buff);
    BlockWrite(outFile, buff, bytesRead, bytesWritten);
  until (bytesRead = 0) or (bytesWritten <> bytesRead);

Code presumes that you won't change size of buffer while processing it. If size of file changes, then you should change last two lines of example code.

Answer 5

I prefer to use tFileStream for this kind of processing. In this example I am assuming there is a constant ArraySize which is set to the size of a single array element. For example, if your "array" is an array of integer then it would be set to:

ArraySize := SizeOf( Integer );

which would set the ArraySize to 4.

Function LoadPos(inFIlename:string;ArrayPos:Int64;var ArrayBuff) : boolean;
var
  fs : tFileStream;
begin
  result := false;
  fs := tFileStream.Create(inFilename,fmOpenRead);
  try
    // seek to the array position
    fs.Seek( ArrayPos * ArraySize, soFromBeginning);
    // load the element
    result := fs.Read( ArrayBuff, ArraySize ) = ArraySize;
  finally
    fs.free;
  end;
end;

The only problem with this approach is it only works for fixed size structures, variable length strings require a different approach.