Python's FTPLib too slow ?

views:

504

answers:

Python's FTPLib too slow ?

I have been playing around with Python's FTP library and am starting to think that it is too slow as compared to using a script file in DOS? I run sessions where I download thousands of data files (I think I have over 8 million right now). My observation is that the download process seems to take five to ten times as long in Python than it does as compared to using the ftp commands in the DOS shell.

Since I don't want anyone to fix my code I have not included any. I am more interested in understanding if my observation is valid or if I need to tinker more with the arguments.

+2 A:

FTPlib may not be the cleanest Python API, I don't think it so bad that it run ten times slower than a DOS shell script.

Unless you do not provide any code to compare, e.g you shell and you python snippet to batch dl 5000 files, I can't see how we can help you.

e-satis 2009-05-07 14:20:38

Thanks for your answer I had no way of knowing what to expect

PyNEwbie 2009-05-07 14:50:51

FTPLib is implemented in Python whereas your "DOS Script" is actually a script which calls a compiled command. Executing this command is probably faster than interpreting Python code. If it is too slow for you, I suggest to call the DOS command from Python using the subprocess module.

Aaron Digulla 2009-05-07 14:35:34

Thanks I did now know about the subprocess module. I have been using SAS to generate a script that I run through the shell but I would like to move everything to Python so I will study the subprocess module

PyNEwbie 2009-05-07 14:52:09

The bottleneck in this can of code is certainly not in the python / C code but more likely during the network process (handshake, ping, dl; etc). This can not explain the low speed he had encountered.

e-satis 2009-05-07 18:20:13

@e-satis: Unlikely since he gets good performance with compiled code for a very large set of measurements.

Aaron Digulla 2009-05-08 08:10:26

+2 A:

The speed problem is probably in your code. FTPlib is not 10 times slower.

nosklo 2009-05-07 15:51:09

Is it slower say I just want to download one file. I played around and could not get it to match the speed on a 1.5 mb file

PyNEwbie 2009-05-07 16:06:59

maybe it can't "match the speed" however it won't be *10 times slower*. Measure the speed difference by downloading many different sized files, either all at once or one at a time, and average the times.

nosklo 2009-05-07 21:12:31

@PyNEwbie: Maybe you should reconsider your decision on not posting the code, I think you're doing something stupid on it.

nosklo 2009-05-07 21:13:29

ansaurus

tags:

views:

answers:

Python's FTPLib too slow ?

related questions