Recursive FTP directory listing in shell/bash with a single session (using cURL or ftp)

Question

I am writing a little shellscript that needs to go through all folders and files on an ftp server (recursively). So far everything works fine using cURL - but it's pretty slow, becuase cURL starts a new session for every command. So for 500 directories, cURL preforms 500 logins.

Does anybody know, whether I can stay logged in using cURL (this would be my favourite solution) or how I can use ftp with only one session in a shell script?

I know how to execute a set of ftp commands and retrieve the response, but for the recursive listing, it has to be a little more dynamic...

Thanks for your help!

Answer 1

Hi, If I recall correctly ncftp supports recursive downloads and uploads with the -R switch maybe you can use that. (dunno if works in batch mode but you can surely script some expect script around it).

Answer 2

if you just want to create a listing of all files and folders, you can use ssh instead. Something like this (but check the documentation on correct usage)

$ ssh user@host "ls -R /path"

Answer 3

You could connect to the ftp server in a manner that it accepts commands from stdin and writes to stdout. Create two named pipes ("fifos", man mkfifo), redirect stdin and stdout of the ftp command each to one of them. Then you can write commands to the stdin-connected-fifo and read them (line-by-line with bash's read for example) from the stdout-fifo. Then use the results to see where you need to send another listing command (and print it or whatever you want to do)

In short: Not something bash scripting is suitable for :) (Until you find a tool that does what you want by itself of course)

Answer 4

@Timo @golemwashere the command is actually 'ncftpls -R'. It will recursively list all the files in a ftp folder.

Answer 5

If it's possible, try usign lftp script:

# lftp script "myscript.lftp"
open your-ftp-host
user username password
cd directory_with_subdirs_u_want_to_list
find
exit

Next thing u need is bash script to run this lftp command and write it to file:

#!/bin/bash
lftp -f myscript.lftp > myOutputFile

myOutputFile now contains the full dump of directories.