Viewing more than 10,000 files on a FTP file server?

Question

I've queued up over 10,000 files to be uploaded to a UNIX based FTP server using a freeware (Windows based) FTP client which as far as i can see has finished without error.

Now, when i view the remote directory (using the Windows software) the output is truncated to 10,000 filenames. This ever occurs when i use the Windows command line FTP tool. Is there a way i can see more than this limit using another piece of software? I just need to confirm all files where indeed uploaded.

Any ideas?

Any information regarding this limit is very welcome.

Answer 1

Please provide more information like specific OS and version. Also please list the names and versions of the ftp clients that you are using when you experience the problem.

A lot of files in a single directory can cause many programs to have problems. I would recommend reducing the number of files in a folder, by putting them in several folders.

The FTP client that you are using probably has an option to look at the raw log and you should be able to use that to determine if it's a problem with the server or your client. The clients could be using a slightly different method to retrieve the list of files.

The story is in the logs.

Answer 2

Sounds like the limit is in the server software. Have you tried changing that?

Just out of curiosity I did a quick check on google for "ftp 10000 file limit", which turns up a heap of results but nothing obvious relating to this unfortunately...

Answer 3

If all you need to do is check the files all uploaded, then why not just try to download the last one?

Answer 4

You should be able to get to the raw FTP interface or somehow otherwise be able to submit single ftp commands directly. If you can, this list gives you some options. . (It's a windows list, but unix should be no worse. The following uses unix ftp commands as best I recall.) The bang (!) is supposed to drop you to a cmd prompt (I'm sure there are security issues involved. If you can somehow "ls | wc" you'll have a count, or perhaps you can "ls" to a file you can ftp back with the file list.

Answer 5

If you have shell/rd access just (ls -al|dir) > filename then do a diff on that file and a similar file from your local machine.

Answer 6

I just created 11,000 files on an FTP server I have access to and then used Firefox to display the directory with a URL beginning ftp://hostname/pub/.... Firefox displays all 11,000 files just fine.

Answer 7

What software is the server running for ftp? If its vsftp, then you might be reaching shell/kernel limits and not the ftp server limits. Also 10K is a tad to much for one directory, seriously might want to consider doing directory spanning ( /1-3 characters of file/4-5 characters of filename/fullname.suffix As long as its consistent, this will make it a lot more manageable to find files.

Answer 8

You can run CrossFTP client, and choose Tools -> Synchornize Directory to compare your local folder and the remote folder. If there is any files missing, you can easily find them.

Some FTP Servers will truncate the file list if it is too big, for example, ProFTPd. You need to check the FTP log message to make sure there is no truncate on the server side as well.

Answer 9

If your unix host supports sftp(most modern ones does) then you should use it, winscp is a free sftp client for windows. You can use it to view/synchronize local&remote directories. It will securely transfer all files with encryption and hash checking