I need to transfer lots of small files to a remote computer within my java program. I was wondering if somebody could suggest the best way to do so... I need to transfer lots of small files and it has to be really fast. Should I use some existing protocol implementation? maybe ftp?
One important thing is that most files would be the same all the time, or the difference would be minor so I was thinking of using git for that purpose. Does anyone have experience with sth like this?
How do you feel about compressing those files and then using ftp? Do you have possibility to decompress on receiver's side?
Git is version control system, there's no need of adding git's files on top of those files of yours, if you will not check out the files later. I'd rather use ftp.
Here's a nice article about java ftp libraries (or you can use a system call to a console ftp client, but I don't like this idea)
Who is receiving the files that you send? another application? You may use a messaging software such as active MQ
or Stick with java net APIs for FTP.
URL url = new URL("ftp://user:password@server/filename;type=i");
URLConnection urlc = url.openConnection();
InputStream is = urlc.getInputStream(); // To download
OutputStream os = urlc.getOutputStream(); // To upload
Wondering why you want to involve git. Does it provide any API to find delta etc? I don't think so. git is a version control system as far as I know.
The most efficient way to transfer lots of small files is as an archive; e.g. ZIP or TAR. If your network is relatively slow, compressing the archive before transmission will make a big difference files. But if the network is really fast, compression may actually make the total time to transfer the files longer. The other factor that makes a big difference is the rate at which the file system can read and (especially) create files.
The Git protocol can be really fast, but it achieves this by only sending files that have changed, and (where possible) sending differences instead of complete files. This approach cannot be used for regular file transfer. Rdist and rsync are older UNIX / Linux tools that take the same (differential) approach to transferring files as Git and other version control systems. They won't help you for the same reasons as Git won't ... in general.
The Apache VFS project is a java library that you can use from your program to copy files between file systems.(E.g. copy local files to FTP/SCP/HTTP.)
Copying can be configured so that only files in the source that are newer than the destination are copied, reducing the amount of data sent.
Links
From your description, rsync is an absolutely perfect fit for your requirements, much superior to the alternatives that have been offered.