What tools, languages, and infrastructure do you use for do batch processing in Linux?
I am looking for something that facilitate the tasks of:
Thanks for any help!
Bash scripts are still used for almost everything. It really depends on what the job is at hand. I tend to resort to Perl for heavy text processing with its wonderful regex support and short syntax for instant hackability. I might use Python if I want to do something more complex that will need to be maintained over time. It really depends. But for the majority of "batch scripting" Bash scripts are used.
Tools languages infrastructure? Vim, bash, cron. Unless I misunderstand your question. For MySQL type tasks I use perl or python.
We use Python.
Process files - the os module does some of this.
Log - the logging module.
Validation - no idea what this means.
Job Controlling (start,strop,reestart a process) - this is complex, ambitious and difficult to define. Some folks use commercial products for this because of the potential auditability issues. Control/M, Tivoli, etc., are good solutions. However, you can build some of this using at and cron.
Mysql Connection - This is done via MySQLdb add-on.
If we're talking about an enterprise implementation, I recommend Load Sharing Facility. This is a system that distributes jobs across a cluster of machines. It allows for queueing of jobs; pausing and stopping jobs; and gives plenty of statistics. You can have one or a few machines be "logon" machines where people submit jobs from and several machines that run the jobs full time.
Have you tried Pentaho Data Integration?
With PDI you can just draw flowcharts instead of programming... It's awesome!
Just kidding.
Bash is your friend. I would also suggest using Python but thats a bit more preference in my opinion. Also if this is something that needs to be accessed or maintained by slightly less ummmm competent individuals you can create an Interface easily using PHP.