In the process of deploying our .net app, I've got about 20 scheduled tasks set up on a server, all of which basically do the same thing: invoke a small .net console app that pulls data from a SQL db and posts it to a web service. Each task invokes a separate copy of the app, each copy having a different lookup ID value in its config file.
All but two of these tasks run reliably every night. Two of the tasks seem to sporadically stop running from time to time, and it's currently a mystery as to why. When they stop running, the scheduled task interface correctly shows their last run date, which is a day or more behind the other tasks, which have continued to run at the scheduled time. The tasks which stopped running do not run again on their own, despite being indicated as scheduled to run every night. There are no errors recorded in the event log or in the scheduled task interface itself. And here's the strangest part to us: If I manually kick off the scheduled task, it runs fine, it invokes the .net console app and everything finishes without anomaly. And then it continues to run fine at its scheduled time, for days or weeks at a time, only to eventually fail, seemingly out of the blue. It appears both tasks always start to fail on the same night.
We're not sure where to go next in troubleshooting this. Any suggestions would be most appreciated.