Cannot append to file when some other process writes to it on *nix systems

Question

I have a very simple piece of code which just writes a small amount of data to a file at some regular interval. Once my program has created the file and appended some data, when I open this file in vim(or any other editor for that matter) and edit it, my process cannot seem to update the file anymore. I do not see any errors being returned from the syscall. I tried tracing the system calls, and did not observe anything weird even while the file is NOT being updated.

Since each process gets its own file table entry which has the current offset, all I was expecting was an output file with data interspersed with writes from the two non-cooperating processes(possibly garbled too). But what I am observing is that my program cannot update the file anymore once any other editor writes to the file.

Couple of other interesting observations

1) When I cat something to the output file, my program can continue to update no problem

2) When multiple instances of my own program are writing to the same file, everything is fine again

I understand that there's mandatory locking to prevent multiple writes, but I am trying to understand whats happening underneath. Also this kind of scenario behaves normally for some loggers (like system log, apache logs etc)

Any ideas to explain this behavior?. Also any hints on how I can debug this further?

My code is pretty simple:

  1 int main(int argc, char** argv)
  2 {
  3     const char* buf;
  4     if(argc < 2)
  5         buf = "test->";
  6     else
  7         buf = argv[1];
  8 
  9     int fd; 
 10     if((fd = open("test.log", O_CREAT|O_WRONLY|O_APPEND, 0644)) == -1) {
 11         perror("Cannot open test.log");
 12         exit(1);
 13     }   
 14 
 15     int num_bytes = strlen(buf), num_bytes_written = -1; 
 16 
 17     while(1) {
 18         if((num_bytes_written = write(fd, buf, num_bytes)) == -1) {
 19             perror("Could not write to fd");
 20         }   
 21         fsync(fd);
 22         sleep(5);
 23     }   
 24 }   

Answer 1

Your vim editor works on a cached version of your file. It modifies this cache while your other programs append to the original file. When you save with vim, you overwrite the original file with the updated cached file and loose all logs.

Answer 2

When the vim(1) editor exits, it's likely replacing the original file with the edited version. Your process is holding the original file open but that file no longer exists in the sense that it's directory entry has been replaced and, so, no process that doesn't already have the file open can access it. Your process is now appending to a file that can't be accessed by any other process. Once your process closes the file, it will be gone for good (unless you run a partition recovery program).