Keep Remote Directory Up-to-date

Question

I absolutely love the Keep Remote Directory Up-to-date feature in Winscp. Unfortunately, I can't find anything as simple to use in OS X or Linux. I know the same thing can theoretically be accomplished using changedfiles or rsync, but I've always found the tutorials for both tools to be lacking and/or contradictory.

I basically just need a tool that works in OSX or Linux and keeps a remote directory in sync (mirrored) with a local directory while I make changes to the local directory.

---

Update

Looking through the solutions, I see a couple which solve the general problem of keeping a remote directory in sync with a local directory manually. I know that I can set a cron task to run rsync every minute, and this should be fairly close to real time.

This is not the exact solution I was looking for as winscp does this and more: it detects file changes in a directory (while I work on them) and then automatically pushes the changes to the remote server. I know this is not the best solution (no code repository), but it allows me to very quickly test code on a server while I develop it. Does anyone know how to combine rsync with any other commands to get this functionality?

Answer 1

How "real-time" do you want the syncing? I would still lean toward rsync since you know it is going to be fully supported on both platforms (Windows, too, with cygwin) and you can run it via a cron job. I have a super-simple bash file that I run on my system (this does not remove old files):

#!/bin/sh
rsync -avrz --progress --exclude-from .rsync_exclude_remote . remote_login@remote_computer:remote_dir    

# options
#   -a  archive
#   -v  verbose
#   -r  recursive
#   -z  compress

Your best bet is to set it up and try it out. The -n (dry-run) option is your friend!

Keep in mind that rsync (at least in cgywin) does not support unicode file names (as of 16 Aug 2008).

Answer 2

You could always use version control, like SVN, so all you have to do is have the server run svn up on a folder every night. This runs into security issues if you are sharing your files publicly, but it works.

If you are using Linux though, learn to use rsync. It's really not that difficult as you can test every command with -n. Go through the man page, the basic format you will want is

rsync [OPTION...] SRC... [USER@]HOST:DEST

the command I run from my school server to my home backup machine is this

rsync -avi --delete ~ me@homeserv:~/School/ >> BackupLog.txt

This takes all of the files in my home directory (~) and uses rsync's archive mode (-a), verbosly (-v), lists all of the changes made (-i), while deleting any files that don't exist anymore (--delete) and puts the in the Folder /home/me/School/ on my remote server. All of the information it prints out (what was copied, what was deleted, etc.) is also appended to the file BackupLog.txt

I know that's a whirlwind tour of rsync, but I hope it helps.

Answer 3

The rsync solutions are really good, especially if you're only pushing changes one way. Another great tool is unison -- it attempts to syncronize changes in both directions. Read more at the Unison homepage.

Answer 4

To detect changed files, you could try fam (file alteration monitor) or inotify. The latter is linux-specific, fam has a bsd port which might work on OS X. Both have userspace tools that could be used in a script together with rsync.

Answer 5

It seems like perhaps you're solving the wrong problem. If you're trying to edit files on a remote computer then you might try using something like the ftp plugin for jedit. http://plugins.jedit.org/plugins/?FTP This ensures that you have only one version of the file so it can't ever be out of sync.

Answer 6

You can also use Fetch as an SFTP client, and then edit files directly on the server from within that. There are also SSHFS (mount an ssh folder as a Volume) options. This is in line with what stimms said - are you sure you want stuff kept in sync, or just want to edit files on the server?

OS X has it's own file notifications system - this is what Spotlight is based upon. I haven't heard of any program that uses this to then keep things in sync, but it's certainly conceivable.

I personally use RCS for this type of thing:- whilst it's got a manual aspect, it's unlikely I want to push something to even the test server from my dev machine without testing it first. And if I am working on a development server, then I use one of the options given above.

Answer 7

Building off of icco's suggestion of SVN, I'd actually suggest that if you are using subversion or similar for source control (and if you aren't, you should probably start) you can keep the production environment up to date by putting the command to update the repository into the post-commit hook.

There are a lot of variables in how you'd want to do that, but what I've seen work is have the development or live site be a working copy and then have the post-commit use an ssh key with a forced command to log into the remote site and trigger an svn up on the working copy. Alternatively in the post-commit hook you could trigger an svn export on the remote machine, or a local (to the svn repository) svn export and then an rsync to the remote machine.

I would be worried about things that detect changes and push them, and I'd even be worried about things that ran every minute, just because of race conditions. How do you know it's not going to transfer the file at the very same instant it's being written to? Stumble across that once or twice and you'll lose all of the time-saving advantage you had by constantly rsyncing or similar.

Answer 8

Will DropBox (http://www.getdropbox.com/) do what you want?

Answer 9

What you want to do for linux remote access is use 'sshfs' - the SSH File System.

# sshfs username@host:path/to/directory local_dir

Then treat it like an network mount, which it is...

A bit more detail, like how to set it up so you can do this as a regular user, on my blog

If you want the asynchronous behavior of winSCP, you'll want to use rsync combined with something that executes it periodically. The cron solution above works, but may be overkill for the winscp use case.

The following command will execute rsync every 5 seconds to push content to the remote host. You can adjust the sleep time as needed to reduce server load.

# while true; do rsync -avrz localdir user@host:path; sleep 5; done

If you have a very large directory structure and need to reduce the overhead of the polling, you can use 'find':

# touch -d 01/01/1970 last; while true; do if [ "`find localdir -newer last -print -quit`" ]; then touch last; rsync -avrz localdir user@host:path; else echo -ne .; fi; sleep 5; done

And I said cron may be overkill? But at least this is all just done from the command line, and can be stopped via a ctrl-C.

kb