LuaSocket FTP always times out

Question

I've had success with LuaSocket's TCP facility, but I'm having trouble with its FTP module. I always get a timeout when trying to retrieve a (small) file. I can download the file just fine using Firefox or ftp in passive mode (on Ubuntu Dapper Linux).

I thought it might be that I need LuaSocket to use passive FTP, but then I found that it seems to do that by default. The file I'm trying to retrieve via FTP can be accessed with passive FTP via other programs on my machine, but not via active mode. I found some talk about "hacking" passive mode support into LuaSocket, and that discussion implies that later versions stopped using passive mode, but my version seems to use passive anyway (I'm using 2.0.1; newest is 2.0.2 and does not appear to have any changes relevant to my use case). I'm a little confused about how that post may relate to my situation, partly because it's very old and LuaSocket's source now bears little resemblance to the code in that discussion).

I've boiled my code down to this:

local ftp = require "socket.ftp"
ftp.TIMEOUT = 10
print(ftp.get("ftp://ftp.us.dell.com/app/dpart.txt"))

This gives me a timeout. I ran it under strace on Linux (same as ptrace on Solaris). Here's an abridged transcript:

socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
fcntl64(3, F_SETFL, O_RDWR|O_NONBLOCK)  = 0
recv(3, "230-Welcome to the Dell FTP site."..., 8192, 0) = 971
send(3, "pasv\r\n", 6, 0)               = 6
recv(3, 0x8089a58, 8192, 0)             = -1 EAGAIN (Resource temporarily unavailable)
select(4, [3], NULL, NULL, {9, 999934}) = 0 (Timeout)

There's another site I tried connecting to, but it has a password which I can't post here, but in that case the results were slightly different...I got trace like the above but with select() succeeding at the end, then this:

recv(3, "227 Entering Passive Mode (123,456,789,0,12,34)\r\n", 8192, 0) = 49
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 4
fcntl64(4, F_SETFL, O_RDWR|O_NONBLOCK)  = 0
connect(4, {sa_family=AF_INET, sin_port=htons(12345), sin_addr=inet_addr("123.456.789.0")}, 16) = -1 EINPROGRESS (Operation now in progress)
select(5, [4], [4], NULL, {9, 999694})  = 0 (Timeout)

Compare this to the trace of my "ftp" program in passive mode (which works fine, though note that it does not set the sockets to nonblocking like LuaSocket does):

socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 6
write(5, "PASV\r\n", 6)                 = 6
read(3, "227 Entering Passive Mode (123,456,789,0,12,34)\r\n", 1024) = 51
connect(6, {sa_family=AF_INET, sin_port=htons(12345), sin_addr=inet_addr("123.456.789.0")}, 16) = 0

So I've tried LuaSocket against these two different FTP sites with different but similar failures. I also tried it from another machine where active FTP works, and it didn't have any better luck there (presumably because LuaSocket is always using passive mode, from what I can tell by reading the source in socket/ftp.lua).

So can anyone here make the LuaSocket two-liner at the top work? Note that on my machine, active FTP to Dell's site doesn't work (I can connect but as soon as I do ls it disconnects), so if you get LuaSocket to work please also note whether active FTP to Dell's site from another program works on your machine.

Answer 1

Hm. It looks like the problem is that LuaSocket uses "pasv" in lower case. I'm going try to figure out a work-around.

---

Hm. Nope, it looks quite elegantly welded shut. The easiest thing to do is probably to copy that particular file to its equivalent place in a hierarchy in an earlier path in LUA_PATH. That is, (usually) make a local copy of the file, e.g. path/to/your/project/socket/ftp.lua.

Then edit the local file:

-    self.try(self.tp:command("user", user or USER))
+    self.try(self.tp:command("USER", user or USER))
-        self.try(self.tp:command("pass", password or PASSWORD))
+        self.try(self.tp:command("PASS", password or PASSWORD))
-    self.try(self.tp:command("pasv"))
+    self.try(self.tp:command("PASV"))
-    self.try(self.tp:command("port", arg))
+    self.try(self.tp:command("PORT", arg))
-    local command = sendt.command or "stor"
+    local command = sendt.command or "STOR"
-    self.try(self.tp:command("cwd", dir))
+    self.try(self.tp:command("CWD", dir))
-    self.try(self.tp:command("type", type))
+    self.try(self.tp:command("TYPE", type))
-    self.try(self.tp:command("quit"))
+    self.try(self.tp:command("QUIT"))

Perversely, a navelnaut expedition using getfenv, getmetatable, etc didn't seem to be worth it. I consider it a serious problem with the design. (of LuaSocket)

It's worth noting that RFC0959 uses all-caps commands. (Probably because it's from the 7-bit ASCII era.)

Answer 2

Note that the server is failing to follow the FTP specification, which states commands are case-insensitive. See RFC959, section 5.3 "The command codes are four or fewer alphabetic characters. Upper and lower case alphabetic characters are to be treated identically. Thus, any of the following may represent the retrieve command: RETR Retr retr ReTr rETr"

Answer 3

This problem is now fixed, with the question and first answer a great help.

Luasocket is correct to RFC 959 (first comment here is not right about upper case, see RFC959 section 5.2)

At least Microsoft FTP server is not compliant. There might be others.

The solution is change pasv to PASV and is a workaround for a command case sensitive server. Details are on the Lua email list, where the archive will be web accessible in a few days.

(edit line 59 of ftp.lua)