views:

67

answers:

1

Hi,

I have a directory structure and files like this

data/
data/a.txt
data/folder/
data/folder/b.txt
data/folder/folder/
data/folder/folder/c.txt
...

a.txt, b.txt, and c.txt are large files that are computer-generated and renewed frequently. They should NOT be backuped -- but I want to backup the directory structure :

data/
data/folder/
data/folder/folder/

How can I do this with rsync and --exclude-from, without specifying every folder, but something like rsync -a data/* --exclude-from=exclude.rsync "" --onlyfoldersandnotfiles""?

Thanks for help !

A: 

To be fairly honest depending on what you want to acomplish a simple ssh would be ok:

find /data/ -type d -exec ssh user@host mkdir /destination/{} \;

Side note: this would probably be slow if u have a huge folder tree


i guess the best you could attempt without the need of external help (ie; extra programs) would be:

  • --exclude "*.*" to exclude all files

  • or more specific --exclude "*.txt"

  • or --exclude 'a.txt' --exclude 'b.txt' --exclude 'c.txt'

  • or like you have already --exclude-from '/my_exclude_list'

my_exclude_list content:

a.txt
b.txt
c.txt

I belive your best way around this would be to use an extra program to create your --exclude-from list with all the files within your directory tree, one way of doing this would be:

find data/ -type f > my_exclude_file

The above command would fine all regular file type and dump it into my_exclude_file which you could then use for your rsync command --exclude-from 'my_exclude_list'

Your final command to this would be:

find /my/directory/tree_data/ -type f > exclude.rsync; rsync -a data/* --exclude-from=exclude.rsync

Prix
Thanks for getting to the problem, however, it does not solve my problem : I want something automatic, and only folders, and not files. The filenames in the examples are taken arbitrarily -- but when I have for example a directory with the name "dir.txt", --exclude "*.txt" will fail as well since it does not copy the directory.Any other idea ? I think a solution would be useful -- keep directory structure, but do not backup the files.
chicama
have updated my answer with how you could acomplish it but for that you would need to use an extra program :)
Prix
Thanks for the update, I appreciate. I perfectly understand this approach. Nevertheless, my exclude.rsync file would grow significantly to a non-human readable format since in my folders there are numerous computer-generated images. Any other suggestions ?
chicama
you can define the find command to ignore image files, for example: `! -name '*.jpg'` to add multiple file types you must define 1 for each for example `! -name '*.jpg' ! -name '*.gif' ! -name '*.png' ! -name '*.tiff'` ofc that is if you know most of the image types you need to exclude, then you can use `echo "*.jpg" >> exclude.rsync` to add a global catch for the images in your file.
Prix
Maybe something like this `for i in $(find /data/ -type d); do echo "mkdir $i" >> send_over.sh; done; chmod +x send_over.sh; rsync -avP /source dest:/dest ` it will generate a file with commands to recreate your directory structure only that you can them send over using rsync and execute
Prix
or you could simplify it with only ssh: `find /data/ -type d -exec ssh user@host mkdir /path/{} \;`
Prix
All your comments are very logical to me, thanks. Even though I miss the adapted switch in rsync, I would like to go with the <b>echo "*.jpg" >> exclude.rsync</b>solution, but I would like to exclude not all jpgs, but only the jpegs of an entire subdirectory tree. Is this possible ? Or will we face the same problem ?
chicama
just change from `*.jpg` to `/data/i_dont_want_the_jpgs_from_this_direcoty/*.jpg`
Prix
Sorry, of course, but this does not effect the entire subdirectory tree. I would like to include /data/i_dont_want_the_jpgs_from_this_direcoty/ and /data/i_dont_want_the_jpgs_from_this_direcoty/subdirectory1/*.jpg, /data/i_dont_want_the_jpgs_from_this_direcoty/subdirectory2/*.jpg etc. --- and since I have hundreds of subdirectories, I need to do this in one command ! So how can I recurse ?
chicama
you would need to use the find like i showed above on the subdirectories you want to produce a list of directories... something like this would do `for i in $(find /data/i_dont_want_the_jpgs_from_this_direcoty -type d); do echo "$i/*.jpg" >> exclude.rsync;`
Prix
After a long voyage, I come up with a really simple solution for the exclude.rsync file : `*.jpg` `*/*.jpg` `*/*/.jpg` `*/*/*/.jpg` `*/*/*/*.jpg` (in 5 lines) excludes all jpgs from subdirectories up to three levels, and more levels work similarly. Prix, what do you think ?
chicama