What's the best way to take flat file logs from a large number of servers and bring them all back to one central location?

Question

These are text-formatted, line-deliminated files written by applications like mysql or apache that are out of my control.

Answer 1

If they're all networked and on the same domain; make a shared folder, and have the programs output their log files to this folder.

Answer 2

You might want to check out Splunk or Q Radar both products that gather log files from many locations and make them searchable.

Answer 3

It really depends on what you are comfortable working with.

You could perform this task with anything ranging from scripts (bash, perl, .bat, etc.) to compiled programs you write yourself to various products (Splunk and Q Radar were mentioned by tschaible) or even something like Microsoft's SSIS.

Hard to say much more until we know what platforms/languages/etc you like to use.

Most likely you would use FTP as a lowest common denominator for pulling the files from the various servers. It can be used by all the options I mentioned.

Seems likely you would want to pull them into a database of some sort to make them searchable.

You could probably get a better answer if you add some more details to your question.

Hope this helps.