How to compare two TXT files before send it to SQL

Question

I have to handle TXT dat files which coming from one embed device, My problem is in that device always sending all captured data but I want to take only difrences between two sending and do calculation on them. After calculation I send it to SQL using bulkinsert function. I want to extract data which is different according to first file I got from device. Lats say that device first time device send data like this in some.dat (ASCII) file

0000199991
0000199321
0000132913
0000232318
0000312898

On second calls to get data from device it is going to return all again (previous and next captured records) something like this

0000199991
0000199321
0000132913
0000232318
0000312898
9992129990
8782999022
2323423456

But this time I do want only to calculate and pass trough data added after first insert. I am trying to make Win Forms app using C# and Visual Studio 2008

Answer 1

One option would be to remember the filesize each time you receive the file, then when you get the new file you can immediately move the file pointer to the position in the file that corresponds to the end of the previous file and read from that point on.

Here is a rough outline of the idea

  long lastPosition = GetLastFilePositionFromDatabase();
  using (FileStream fs = new FileStream(...))
  {
    // Seek to the last position, this is zero the first time
    fs.Seek(lastFilePosition, SeekOrigin.Begin);

    // Process your file from the current position
    ProcessFile(fs);

    // Once you reach the end of the file, save this position so 
    // for use with the next file
    SaveLastFilePositionToDatabase(fs.Position);
  }

Answer 2

You can do this using LINQ:

string[] addedLines = File.ReadAllLines(secondPath)
                          .Except(File.ReadAllLines(firstPath))
                          .ToArray();

Note that this will be slow for large files.

For large files, replace ReadAllLines with the following method: (In .Net 4, you can use File.ReadLines instead)

static IEnumerable<string> EnumerateLines(string path) {
    using(var reader = File.OpenText(path)) {
        string line;
        while(null != (line = reader.ReadLine())
            yield return line;
    }
}

Answer 3

Would this work for you?

string dataAddedAfterFirstInsert = secondString.SubString(firstString.Length, secondString.Length)