views:

819

answers:

5

I have a binary log file with streaming data from a sensor (Int16).
Every 6 seconds, 6000 samples of type Int16 are added, until the sensor is disconnected.

I need to poll this file on regular intervals, continuing from last position read. Is it better to a) keep a filestream and binary reader open and instantiated between readings b) instantiate filestream and binary reader each time I need to read (and keep an external variable to track the last position read) c) something better?

EDIT: Some great suggestions so far, need to add that the "server" app is supplied by an outside source vendor and cannot be modified.

+2  A: 

If it's always adding the same amount of data, it may make sense to reopen it. You might want to find out the length before you open it, and then round down to the whole number of "sample sets" available, just in case you catch it while it's still writing the data. That may mean you read less than you could read (if the write finishes between you checking the length and starting the read) but you'll catch up next time.

You'll need to make sure you use appropriate sharing options so that the writer can still write while you're reading though. (The writer will probably have to have been written with this in mind too.)

Jon Skeet
The writer can't be modified, and it is written in unmanaged code (not sure what language, or what options it is using to write the data). Since it is updating by 12,000 bytes in 6 second intervals, am I safe always tailing by 6 seconds, even if it is writing while I am reading a previous block (FileShare.ReadWrite)?
mikeh
@mikeh: That will depend on how the writer has opened the file, but it's worth trying. I wouldn't do anything which relied on staying absolutely in sync with the writer though.
Jon Skeet
+2  A: 

Can you use MemoryMappedFiles?

If you can, mapping the file in memory and sharing it between processes you will be able to read the data by simply incrementing the offset for your pointer each time.

If you combine it with an event you can signal your reader when he can go in an read the information. There will be no need to block anything as the reader will always read "old" data which has already been written.

Jorge Córdoba
A: 

you could also stream the data into a database, rather than a file as another alternative, then you wouldn't have to worry about file locking.

but if you're stuck with the file method, you may want to close the file each time you read data from it; it depends alot on how complicated the process writing to the file is going to be, and whether it can detect a file locking operation and respond appropriately without crashing horribly.

Phill
+1  A: 

I think that (a) is the best because:

  • Current Position will be incremented as you read and you don't need to worry about to store it somewhere;
  • You don't need to open it and seek required position (it shouldn't be much slower to reopen but keeping it open gives OS some hints for optimization I believe) each time you poll it;
  • Other solutions I can think out requires PInvokes to system interprocess synchronisation primitives. And they won't be faster than file operations already in framework.

You just need to set proper FileShare flags:

Just for example:

Server:

using(var writer = new BinaryWriter(new FileStream(@"D:\testlog.log", FileMode.Append, FileAccess.Write, FileShare.Read)))
{
    int n;
    while(Int32.TryParse(Console.ReadLine(), out n))
    {
        writer.Write(n);
        writer.Flush(); // write cached bytes to file
    }
}

Client:

using (var reader = new BinaryReader(new FileStream(@"D:\testlog.log", FileMode.Open, FileAccess.Read, FileShare.ReadWrite)))
{
    string s;
    while (Console.ReadLine() != "exit")
    {
        // allocate buffer for new ints
        Int32[] buffer = new Int32[(reader.BaseStream.Length - reader.BaseStream.Position) / sizeof(Int32)];

        Console.WriteLine("Stream length: {0}", reader.BaseStream.Length);
        Console.Write("Ints read: ");
        for (int i = 0; i < buffer.Length; i++)
        {
            buffer[i] = reader.ReadInt32();
            Console.Write((i == 0 ? "" : ", ") + buffer[i].ToString());
        }
        Console.WriteLine();
    }
}
Regent
+1  A: 

I would recommend using pipes, they act just like files, except stream data directly between applications, even if the apps run on different PCs (though this is really only an option if you are able to change both applications). Check it out under the "System.IO.Pipes" namespace.

P.S. You would use a "named" pipe for this (pipes are supported in 'c' as well, so basically any half decent programming language should be able to implement them)

Grant Peters