Handling very large SFTP uploads - Cocoa

Question

Hello everyone,

I'm working on a small free Cocoa app that involves some SFTP functionality, specifically working with uploads. The app is nearing completion however I've run into a pretty bad issue with regards to uploading folders that contain a lot of files.

I'm using ConnectionKit to handle the uploads:

CKTransferRecord * record;
record = [connection recursivelyUpload:@"/Users/me/large-folder" 
                                    to:@"/remote/directory"];

This works fine for most files and folders. Although in this case @"/Users/me/large-folder" has over 300 files in it. Calling this method spins my CPU up to 100% for about 30 seconds and my application is unresponsive (mac spinning ball). After the 30 seconds my upload is queued and works fine, but this is hardly ideal. Obviously whatever is enumerating these files is causing my app to lock up until it's done.

Not really sure what to do about this. I'm open to just about any solution - even using a different framework, although I've done my research and ConnectionKit seems to be the best of what's out there.

Any ideas?

Answer 1

Use Shark. Start sampling, start downloading, and as soon as the hang ends, stop sampling.

If the output confirms that the problem is in ConnectionKit, you have two choices:

1. Switch to something else.
2. Contribute a patch that makes it not hang.

The beauty of open-source is that #2 is possible. It's what I recommend. Then, not only will you have a fast ConnectionKit, but once the maintainers accept your patch, everyone else who uses CK can have one too.

And if Shark reveals that the problem is not in ConnectionKit (rule #2 of profiling: you will be surprised), then you have Shark's guidance on how to fix your app.

Answer 2

There is already an open source SFTP application that you may not be aware of. It's called FUGU (http://rsug.itd.umich.edu/software/fugu/). You may want to browse that source code and see if it helps you fix your problem.

Answer 3

Since the problem is almost certainly on the enumeration, you'll likely need to move the enumeration into an asynchronous action. Most likely they're using NSFileManager -enumeratorAtPath: for this. If that's the main problem, then the best solution is likely to move that work onto its own thread. Given the very long time involved, I suspect they're actually reading the files during enumeration. The solution to that is to read the files lazily, right before uploading.

Peter is correct that Shark is helpful, but after being a long-time Shark fan, I've found that Instruments tends to give more useable answers more quickly. You can more easily add a disk I/O and memory allocation track to your CPU Sampler using Instruments.

If you're blocking just one core at 100%, I recommend setting Active Thread to "Main Thread" and setting Sample Perspective to "All Sample Counts." If you're blocking all your cores at 100%, I recommend setting Active Thread to "All Threads" and Sample Perspective to "Running Sample Times."