I am writing a client for a backup server. My client schedules some folders for backup. (ex: every friday at hour X). I am using for scheduling cron4j (a port of Linux cron to java).
Everything works nice until i schedule at the same time multiple upload jobs, then due to the multiple threads it gets messy.
Can anyone help me with a solution for multi-threaded uploads to a ftp server ? (each thread has the job to upload a directory on the Ftp server).
Can you explain what you mean when you say it gets a bit "messy"? Any particular symptoms?
Anyway looking at this from a clean sheet I'd say you want to throttle the number of concurrent uploads you're performing at one time otherwise you're likely to hit some kind of limit in terms of the number of connections the FTP server will allow a single client, or the number of connections the client operating system will allow. Your application is likely to be bound more by bandwidth than by CPU so running too many threads may be counter-productive. You'll essentially be uploading more items at once but with less throughput.
I presume you application has a set of tasks which cron4j off, these tasks then take a directory and upload it via FTP, does this sound about right?
If so I'd suggest splitting this into a couple of stages, first cron4j kicks off tasks which constructs a runnable object which when executed will perform the FTP upload instead of the task performing the upload itself. Place this runnable in in a queue (a blocking queue from java.util.concurrent would be a good idea). You then have a few instances of another task running in a thread pool, these tasks will take jobs off the queue and execute them (experiment with the number of threads in the pool to see what gives you good throughput). This will give you a set of competing consumers which you can limit to the effective number of concurrent uploads (e.g. 4 at a time).
As is noted in the comments the SEDA pattern sounds quite interesting. If your going to do any concurrent programming in Java I'd recommend getting a copy of "Concurrent Programming in Java: Design Principles and Patterns" by Doug Lea, it doesn't include the Java 5 stuff but most of that is based on what's in the book anyway. He clearly explains a lot of issues about thread safety and how to write good Java Code.
ipworks FTP class is thread safe. And you must read Java Concurrency in Practice by Brian Goetz ...
public class ServerConnection extends Thread {
// the connection to the ftp server
private final Ftp connection;
or you can implement Runnable and use an executorService if you don't like to manage the thread yourself ...