views:

2434

answers:

4

I monitor files that are dropped on a ftp via filesystem watcher then move to another dir. Now I trigger the copy off the create event of the filesystem watcher but obviously in the case of ftp the create is just a stub file and the data comes in and fills the file as it uploads till complete. Anyone have an elegant solution for this, or do I have to do what I think I have to do

1 wait till last access time is about n ms in past before I copy
2 throw a control file in there to state that that file is done being copied, then delete control file
3 pound the crap out of it
+2  A: 

Wait until you can exclusively open the file- I wouldn't go as far to say that it is a nice solution, but probably the safest strategy in the circumstances.

RichardOD
cant remember the syntax for checking if an exclusive open? what is it?
Brandon Grossutti
i decided to bite the bullet and make my own implementation of filesystemwatcher for ftp, will post when done
Brandon Grossutti
You want the value of None on this enum - http://msdn.microsoft.com/en-us/library/system.io.fileshare.aspx
RichardOD
Surely .Read is the most appropriate? As Cheeso explains in this answer - "FileShare.Read says 'future openers can open the file only with Read access'": http://stackoverflow.com/questions/897796/how-do-i-open-an-already-opened-file-with-a-net-streamreader/898017#898017
rohancragg
@Rohancragg. Chances are you don't need to share the file. I know that None works, I don't know that Read works (I'd have to test it to check). This answer also does the same as me- http://stackoverflow.com/questions/278075/filewatcher-created-event
RichardOD
+2  A: 

this is a very naive implementation but it suits my purposes, i have seen enough people with this problem on the web so decided to contribute. The implementation is fairly specific to my needs I almost completely unconcerned with changed events given the nature of my problem but people can throw their own code in there if they need to do something different, it really the created which cause the most issues. I havent fully tested this but at first write it looks good

using System;
using System.Collections.Generic;
using System.IO;
using System.Timers;

namespace FolderSyncing
{
    public class FTPFileSystemWatcher 
    {
        private readonly string _path;
        public event FileSystemEventHandler FTPFileCreated;
        public event FileSystemEventHandler FTPFileDeleted;
        public event FileSystemEventHandler FTPFileChanged;

        private Dictionary<string, LastWriteTime> _createdFilesToCheck;
        private readonly object _lockObject = new object();
        private const int _milliSecondsSinceLastWrite = 5000;
        private const int _createdCheckTimerInterval = 2000;

        private readonly FileSystemWatcher _baseWatcher;

        public FTPFileSystemWatcher(string path, string Filter)
        {
            _path = path;
            _baseWatcher = new FileSystemWatcher(path,Filter);
            SetUpEventHandling();
        }

        public FTPFileSystemWatcher(string path)
        {
            _path = path;
            _baseWatcher = new FileSystemWatcher(path);
            SetUpEventHandling();
        }

        private void SetUpEventHandling()
        {
            _createdFilesToCheck = new Dictionary<string, LastWriteTime>();
            Timer copyTimer = new Timer(_createdCheckTimerInterval);
            copyTimer.Elapsed += copyTimer_Elapsed;
            copyTimer.Enabled = true;
            copyTimer.Start();
            _baseWatcher.EnableRaisingEvents = true;
            _baseWatcher.Created += _baseWatcher_Created;
            _baseWatcher.Deleted += _baseWatcher_Deleted;
            _baseWatcher.Changed += _baseWatcher_Changed;
        }

        void copyTimer_Elapsed(object sender, ElapsedEventArgs e)
        {
            lock (_lockObject)
            {
                Console.WriteLine("Checking : " + DateTime.Now);
                DateTime dateToCheck = DateTime.Now;
                List<string> toRemove = new List<string>();
                foreach (KeyValuePair<string, LastWriteTime> fileToCopy in _createdFilesToCheck)
                {
                    FileInfo fileToCheck = new FileInfo(_path + fileToCopy.Key);
                    TimeSpan difference = fileToCheck.LastWriteTime - fileToCopy.Value.Date;
                    fileToCopy.Value.Update(fileToCopy.Value.Date.AddMilliseconds(difference.TotalMilliseconds));
                    if (fileToCopy.Value.Date.AddMilliseconds(_milliSecondsSinceLastWrite) < dateToCheck)
                    {
                        FileSystemEventArgs args = new FileSystemEventArgs(WatcherChangeTypes.Created, _path, fileToCopy.Key);
                        toRemove.Add(fileToCopy.Key);
                        InvokeFTPFileCreated(args);
                    }
                }
                foreach (string removal in toRemove)
                {
                    _createdFilesToCheck.Remove(removal);
                }
            }
        }



        void _baseWatcher_Changed(object sender, FileSystemEventArgs e)
        {
            InvokeFTPFileChanged(e);
        }

        void _baseWatcher_Deleted(object sender, FileSystemEventArgs e)
        {
            InvokeFTPFileDeleted(e);
        }

        void _baseWatcher_Created(object sender, FileSystemEventArgs e)
        {
            if (!_createdFilesToCheck.ContainsKey(e.Name))
            {
                FileInfo fileToCopy = new FileInfo(e.FullPath);
                _createdFilesToCheck.Add(e.Name,new LastWriteTime(fileToCopy.LastWriteTime));
            }
        }

        private void InvokeFTPFileChanged(FileSystemEventArgs e)
        {
            FileSystemEventHandler Handler = FTPFileChanged;
            if (Handler != null)
            {
                Handler(this, e);
            }
        }

        private void InvokeFTPFileDeleted(FileSystemEventArgs e)
        {
            FileSystemEventHandler Handler = FTPFileDeleted;
            if (Handler != null)
            {
                Handler(this, e);
            }
        }

        private void InvokeFTPFileCreated(FileSystemEventArgs e)
        {
            FileSystemEventHandler Handler = FTPFileCreated;
            if (Handler != null)
            {
                Handler(this, e);
            }
        }
    }

    public class LastWriteTime
    {
        private DateTime _date;

        public DateTime Date
        {
            get { return _date; }
        }

        public LastWriteTime(DateTime date)
        {
            _date = date;
        }

        public void Update(DateTime dateTime)
        {
            _date = dateTime;
        }



    }
}
Brandon Grossutti
A: 

here's an implemtation to keep in sync

using System;
using System.Configuration;
using System.IO;
using System.Threading;

namespace FolderSyncing
{
    public class FolderSync
    {
        private readonly string masterDirectoryPath;
        private readonly string slaveDirectoryPath;

        public FolderSync()
        {
            masterDirectoryPath = ConfigurationManager.AppSettings.Get("MasterDirectory");
            slaveDirectoryPath = ConfigurationManager.AppSettings.Get("SlaveDirectory");

            if (Directory.Exists(masterDirectoryPath) && Directory.Exists(slaveDirectoryPath))
            {
                FTPFileSystemWatcher watcher = new FTPFileSystemWatcher(masterDirectoryPath);
                watcher.FTPFileChanged += watcher_FTPFileChanged;
                watcher.FTPFileCreated += watcher_FTPFileCreated;
                watcher.FTPFileDeleted += watcher_FTPFileDeleted;
            }
            else
            {
                Console.WriteLine("Directories were not located check config paths");
            }

        }

        void watcher_FTPFileDeleted(object sender, FileSystemEventArgs e)
        {
            DeleteFile(slaveDirectoryPath + e.Name, 5, 1);
        }

        void watcher_FTPFileCreated(object sender, FileSystemEventArgs e)
        {
            CopyFile(e.Name,5,1);
        }

        void watcher_FTPFileChanged(object sender, FileSystemEventArgs e)
        {

        }

        private void DeleteFile(string fullPath, int attempts, int attemptNo)
        {
            if (File.Exists(fullPath))
            {
                try
                {
                    File.Delete(fullPath);
                    Console.WriteLine("Deleted " + fullPath);
                }
                catch (Exception)
                {
                    if (attempts > attemptNo)
                    {
                        Console.WriteLine("Failed deleting  " + fullPath + "trying again "+ attemptNo.ToString()+ " of "+attempts);
                        Thread.Sleep(500);
                        DeleteFile(fullPath, attempts, attemptNo + 1);
                    }
                    else
                    {
                        Console.WriteLine("Critically Failed deleting  " + fullPath);
                    }
                }
            }
        }

        private void CopyFile(string fileName,int attempts, int attemptNo)
        {
            string masterFile = masterDirectoryPath + fileName;
            string slaveFile = slaveDirectoryPath + fileName;
            if (File.Exists(masterFile))
            {
                try
                {
                    File.Copy(masterFile,slaveFile);
                    Console.WriteLine("Copied  " + masterFile);
                }
                catch (Exception)
                {
                    if (attempts > attemptNo)
                    {
                        Console.WriteLine("Failed copying  " + masterFile + "trying again " + attemptNo.ToString() + " of " + attempts);
                        Thread.Sleep(500);
                        CopyFile(fileName, attempts, attemptNo + 1);
                    }
                    else
                    {
                        Console.WriteLine("Critically Failed copying  " + masterFile);
                    }
                }
            }
       }
    }
}
Brandon Grossutti
A: 

Have the source upload a stub file directly after the data file completes, and have your FileSystemWatcher watch for the stub file. For example, if the data file name is mydata_01234 then the stub woulb be mydata_01234_stub. The FileSystemWatcher should then have a mask of "*_stub". You then know the data file name by stripping off the "_stub" suffix. And the stub file cannot be uploaded until after the data file completes, so the data file will be free.

If the stub files are just one byte each you should be able to remove them after whatever operation you're performing with the data file without issue. If your operations are particularly fast, add a 100 ms sleep before deleting the stub.