I want to do an HTTP POST that looks like an HMTL form posted from a browser. Specifically, post some text fields and a file field.
Posting text fields is straightforward, there's an example right there in the net/http rdocs, but I can't figure out how to post a file along with it.
Net::HTTP doesn't look like the best idea. curb is looking good.
Ok, here's a simple example using curb.
require 'yaml'
require 'curb'
# prepare post data
post_data = fields_hash.map { |k, v| Curl::PostField.content(k, v.to_s) }
post_data << Curl::PostField.file('file', '/path/to/file'),
# post
c = Curl::Easy.new('http://localhost:3000/foo')
c.multipart_form_post = true
c.http_post(post_data)
# print response
y [c.response_code, c.body_str]
Posting files is nasty. You need to set the content type of your post to multipart/form-data and then your body must contain multiple 'parts'. Each part is like it's own little request with it's own headers and body, and is separated by a boundary which must be of the format --RANDOM_STRING--
Here's a link to some code which does it in python (best example I could find, as I only do it in C# myself)
curb looks like a great solution, but in case it doesn't meet your needs, you can do it with Net::HTTP. A multipart form post is just a carefully-formatted string with some extra headers. It seems like every Ruby programmer who needs to do multipart posts ends up writing their own little library for it, which makes me wonder why this functionality isn't built-in. Maybe it is... Anyway, for your reading pleasure, I'll go ahead and give my solution here. This code is based off of examples I found on a couple of blogs, but I regret that I can't find the links anymore. So I guess I just have to take all the credit for myself...
The module I wrote for this contains one public class, for generating the form data and headers out of a hash of String and File objects. So for example, if you wanted to post a form with a string parameter named "title" and a file parameter named "document", you would do the following:
data, headers = Multipart::Post.prepqre_query("title" => my_string, "document" => my_file)
Then you just do a normal POST with Net::HTTP:
http = Net::HTTP.new(upload_uri.host, upload_uri.port)
res = http.start {|con| con.post(upload_uri.path, data, headers) }
Or however else you want to do the POST. The point is that Multipart returns the data and headers that you need to send. And that's it! Simple, right? Here's the code for the Multipart module (you need the mime-types gem):
# Takes a hash of string and file parameters and returns a string of text
# formatted to be sent as a multipart form post.
#
# Author:: Cody Brimhall <mailto:[email protected]>
# Created:: 22 Feb 2008
require 'rubygems'
require 'mime/types'
require 'cgi'
module Multipart
VERSION = "1.0.0" unless const_defined?(:VERSION)
# Formats a given hash as a multipart form post
# If a hash value responds to :string or :read messages, then it is
# interpreted as a file and processed accordingly; otherwise, it is assumed
# to be a string
class Post
# We have to pretend like we're a web browser...
USERAGENT = "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-us) AppleWebKit/523.10.6 (KHTML, like Gecko) Version/3.0.4 Safari/523.10.6" unless const_defined?(:USERAGENT)
BOUNDARY = "0123456789ABLEWASIEREISAWELBA9876543210" unless const_defined?(:BOUNDARY)
CONTENT_TYPE = "multipart/form-data; boundary=#{ BOUNDARY }" unless const_defined?(:CONTENT_TYPE)
HEADER = { "Content-Type" => CONTENT_TYPE, "User-Agent" => USERAGENT } unless const_defined?(:HEADER)
def self.prepare_query(params)
fp = []
params.each do |k, v|
# Are we trying to make a file parameter?
if v.respond_to?(:path) and v.respond_to?(:read) then
fp.push(FileParam.new(k, v.path, v.read))
# We must be trying to make a regular parameter
else
fp.push(StringParam.new(k, v))
end
end
# Assemble the request body using the special multipart format
query = fp.collect {|p| "--" + BOUNDARY + "\r\n" + p.to_multipart }.join("") + "--" + BOUNDARY + "--"
return query, HEADER
end
end
private
# Formats a basic string key/value pair for inclusion with a multipart post
class StringParam
attr_accessor :k, :v
def initialize(k, v)
@k = k
@v = v
end
def to_multipart
return "Content-Disposition: form-data; name=\"#{CGI::escape(k)}\"\r\n\r\n#{v}\r\n"
end
end
# Formats the contents of a file or string for inclusion with a multipart
# form post
class FileParam
attr_accessor :k, :filename, :content
def initialize(k, filename, content)
@k = k
@filename = filename
@content = content
end
def to_multipart
# If we can tell the possible mime-type from the filename, use the
# first in the list; otherwise, use "application/octet-stream"
mime_type = MIME::Types.type_for(filename)[0] || MIME::Types["application/octet-stream"][0]
return "Content-Disposition: form-data; name=\"#{CGI::escape(k)}\"; filename=\"#{ filename }\"\r\n" +
"Content-Type: #{ mime_type.simplified }\r\n\r\n#{ content }\r\n"
end
end
end
I like RestClient. It encapsulates net/http with cool features like multipart form data:
require 'rest_client'
RestClient.post 'http://localhost:3000/foo', fields_hash.merge(:file => File.new('/path/to/file'))
It also supports streaming.
gem install rest-client will get you started.
Well the solution with NetHttp has a drawback that is when posting big files it loads the whole file into memory first.
After playing a bit with it I came up with the following solution:
class Multipart
def initialize( file_names )
@file_names = file_names
end
def post( to_url )
boundary = '----RubyMultipartClient' + rand(1000000).to_s + 'ZZZZZ'
parts = []
streams = []
@file_names.each do |param_name, filepath|
pos = filepath.rindex('/')
filename = filepath[pos + 1, filepath.length - pos]
parts << StringPart.new ( "--" + boundary + "\r\n" +
"Content-Disposition: form-data; name=\"" + param_name.to_s + "\"; filename=\"" + filename + "\"\r\n" +
"Content-Type: video/x-msvideo\r\n\r\n")
stream = File.open(filepath, "rb")
streams << stream
parts << StreamPart.new (stream, File.size(filepath))
end
parts << StringPart.new ( "\r\n--" + boundary + "--\r\n" )
post_stream = MultipartStream.new( parts )
url = URI.parse( to_url )
req = Net::HTTP::Post.new(url.path)
req.content_length = post_stream.size
req.content_type = 'multipart/form-data; boundary=' + boundary
req.body_stream = post_stream
res = Net::HTTP.new(url.host, url.port).start {|http| http.request(req) }
streams.each do |stream|
stream.close();
end
res
end
end
class StreamPart
def initialize( stream, size )
@stream, @size = stream, size
end
def size
@size
end
def read ( offset, how_much )
@stream.read ( how_much )
end
end
class StringPart
def initialize ( str )
@str = str
end
def size
@str.length
end
def read ( offset, how_much )
@str[offset, how_much]
end
end
class MultipartStream
def initialize( parts )
@parts = parts
@part_no = 0;
@part_offset = 0;
end
def size
total = 0
@parts.each do |part|
total += part.size
end
total
end
def read ( how_much )
if @part_no >= @parts.size
return nil;
end
how_much_current_part = @parts[@part_no].size - @part_offset
how_much_current_part = if how_much_current_part > how_much
how_much
else
how_much_current_part
end
how_much_next_part = how_much - how_much_current_part
current_part = @parts[@part_no].read(@part_offset, how_much_current_part )
if how_much_next_part > 0
@part_no += 1
@part_offset = 0
next_part = read ( how_much_next_part )
current_part + if next_part
next_part
else
''
end
else
@part_offset += how_much_current_part
current_part
end
end
end
Here is my solution after trying other ones available on this post, I'm using it to upload photo on TwitPic:
def upload(photo)
`curl -F media=@#{photo.path} -F username=#{@username} -F password=#{@password} -F message='#{photo.title}' http://twitpic.com/api/uploadAndPost`
end
I had the same problem (need to post to jboss web server). Curb works fine for me, except that it caused ruby to crash (ruby 1.8.7 on ubuntu 8.10) when I use session variables in the code.
I dig into the rest-client docs, could not find indication of multipart support. I tried the rest-client examples above but jboss said the http post is not multipart.
Pedro's reply above (1st position) is technically wrong, but I don't have the karma to add a comment to his post to alert readers. :)
At the moment only technoweenie's rest-client fork does multi-part sends for rest-client. It hasn't been pulled into the master rest-client branch at the official rest-client GitHub.
Another alternative is httpclient:
require 'httpclient' # gem install httpclient
HTTPClient.post 'http://www.example.com', { :file => File.new('/home/foo/bar.jpg') }
there's also nick sieger's multipart-post to add to the long list of possible solutions.
restclient did not work for me until I overrode create_file_field in RestClient::Payload::Multipart.
It was creating a 'Content-Disposition: multipart/form-data' in each part where it should be ‘Content-Disposition: form-data’.
http://www.ietf.org/rfc/rfc2388.txt
My fork is here if you need it: [email protected]:kcrawford/rest-client.git
I can't say enough good things about Nick Sieger's multipart-post library.
It adds support for multipart posting directly to Net::HTTP, removing your need to manually worry about boundaries or big libraries that may have different goals than your own.
Here is a little example on how to use it from the README:
require 'net/http/post/multipart'
url = URI.parse('http://www.example.com/upload')
File.open("./image.jpg") do |jpg|
req = Net::HTTP::Post::Multipart.new url.path,
"file" => UploadIO.new(jpg, "image/jpeg", "image.jpg")
res = Net::HTTP.start(url.host, url.port) do |http|
http.request(req)
end
end
You can check out the library here: http://github.com/nicksieger/multipart-post
or install it with:
$ sudo gem install multipart-post
Another approach is not to use multipart at all, but to embed the file's binary content into your resource's XML representation. The XML then goes into the request body. This is easy to do using ActiveResource with no new dependencies.
I've created a plugin to upload/download Paperclip attachments this way in Rails 3:
http://github.com/nragaz/encoded_attachment
Appreciate your feedback.