Google App Engine urlfetch to POST files

Question

I am trying to send a file to torrage.com from an app in GAE. the file is stored in memory after being received from a user upload.

I would like to be able to post this file using the API available here: http://torrage.com/automation.php but i am having some problems undestanding how the body of the post should be encoded, the most i got from the API is a "file empty" message.

Answer 1

Why not just use Python's urllib2 module to create a POST request, like they show in an example for PHP. It would be something like this:

import urrlib, urllib2
data = (
        ('name', 'torrent'), 
        ('type', 'application/x-bittorrent'),
        ('file', '/path/to/your/file.torrent'),
)
request = urllib2.urlopen('http://torrage.com/autoupload.php', urllib.urlencode(data))

Answer 2

I find torrage's API docs on the POST interface (as opposed to the SOAP one) pretty confusing and conflicting with the sample C code they also supply. It seems to me that in their online example of PHP post they are not sending the file's contents (just like @kender's answer above is not sending it) while they ARE sending it in the SOAP examples and in the example C code.

The relevant part of the C sample (how they compute the headers that you'd be passing to urlfetch.fetch) is:

  snprintf(formdata_header, sizeof(formdata_header) - 1,
    "Content-Disposition: form-data; name=\"torrent\"; filename=\"%s\"\n"
    "Content-Type: " HTTP_UPLOAD_CONTENT_TYPE "\n"
    "\n",
    torrent_file);
  http_content_len = 2 + strlen(content_boundary) + 1 + strlen(formdata_header) + st.st_size + 1 + 2 + strlen(content_boundary) + 3;
  LTdebug("http content len %u\n", http_content_len);
  snprintf(http_req, sizeof(http_req) - 1, 
    "POST /%s HTTP/1.1\n"
    "Host: %s\n"
    "User-Agent: libtorrage/" LTVERSION "\n"
    "Connection: close\n"
    "Content-Type: multipart/form-data; boundary=%s\n"
    "Content-Length: %u\n"
    "\n",
    cache_uri, cache_host, content_boundary, http_content_len);

"application/x-bittorrent" is the HTTP_UPLOAD_CONTENT_TYPE. st.st_size is the number of bytes in the memory buffer with all the file's data (the C sample reads that data from file, but it doesn't matter how you got it into memory, as long as you know its size). content_boundary is a string that's NOT present in the file's contents, they build it as "---------------------------%u%uLT" with each %u substituted by a random number (repeating until that string hits upon two random numbers that make it not present in the file). Finally, the post body (after opening the HTTP socket and sending the other headers) they write as follows:

  if (write_error == 0) if (write(sock, "--", 2) <= 0) write_error = 1;
  if (write_error == 0) if (write(sock, content_boundary, strlen(content_boundary)) <= 0) write_error = 1;
  if (write_error == 0) if (write(sock, "\n", 1) <= 0) write_error = 1;
  if (write_error == 0) if (write(sock, formdata_header, strlen(formdata_header)) <= 0) write_error = 1;
  if (write_error == 0) if (write(sock, filebuf, st.st_size) <= 0) write_error = 1;
  if (write_error == 0) if (write(sock, "\n--", 3) <= 0) write_error = 1;
  if (write_error == 0) if (write(sock, content_boundary, strlen(content_boundary)) <= 0) write_error = 1;
  if (write_error == 0) if (write(sock, "--\n", 3) <= 0) write_error = 1;

where filebuf is the buffer with the file's contents.

Hardly sharp and simple, but I hope there's enough info here to work out a way to build the arguments for a urlfetch.fetch (building them for a urllib.urlopen would be just as hard, since the problem is the scarcity of documentation about exactly what headers and what content and how encoded you need -- and that not-well-documented info needs to be reverse engineered from what I'm presenting here, I think).

Alternatively, it may be possible to hack a SOAP request via urlfetch; see here for Carson's long post of his attempts, difficulties and success in the matter. And, good luck!

Answer 3

Judging from the C code, it's using the "multipart/form-data" format, which is very complex and it's very easy to get something wrong. I wouldn't hand-code the post body like that.

I used the function from this blog and it worked for me from stand-alone program. You might want give it a try in app engine,

http://peerit.blogspot.com/2007/07/multipartposthandler-doesnt-work-for.html