I have to write an console application to import log files daily to database. The log files and the database are on different servers. And then an web application will search through this database.
I'm considering that should I migrate importing application to this web app or I config this app as a job in SQL Server. (as mentioned earlier, importing works must be done daily, at specific time).
In case that I should migrate to the web app, how can I schedule this effectively? (i'm using C#)
Server A has the logs. Server B will process them
-Set up FTP on server B
-Write batch file on server A to FTP log file to server B
-Set up this batch file as a scheduled task on Server A
-Write console app to look for this file in FTP dir and import to DB.
-Set up this as a scheduled task on server B (say 1 hour after other schedule).
One technique is to schedule a callback on startup by adding a [HttpContext] cache entry that expires at whatever time the next run is due. Then one can subscribe to the 'cache entry being removed' callback and do the import then. Upon completion, add the dummy entry back into the cache.
The problem with any approach like this is of course how to notify the operartor of failures and/or manage what happens if a retry is needed for any reason.
Not sure what kind of "logs" you are dealing with, but MS Log Parser might be of some help to you. You can download it here