URL encode in Erlang

Question

I'm using erlang http:request to post some data to a remote service. I have the post working but the data in the body() of the post comes through as is, without any url encoding which causes the post to fail when parsed by the remote service.

Is there a function in Erlang that is similar to CGI.escape in Ruby for this purpose?

Answer 1

AFAIK there's no URL encoder in the standard libraries. Think I 'borrowed' the following code from YAWS or maybe one of the other Erlang web servers:

% Utility function to convert a 'form' of name-value pairs into a URL encoded
% content string.

urlencode(Form) ->
    RevPairs = lists:foldl(fun({K,V},Acc) -> [[quote_plus(K),$=,quote_plus(V)] | Acc] end, [],Form),
    lists:flatten(revjoin(RevPairs,$&,[])).

quote_plus(Atom) when is_atom(Atom) ->
    quote_plus(atom_to_list(Atom));

quote_plus(Int) when is_integer(Int) ->
    quote_plus(integer_to_list(Int));

quote_plus(String) ->
    quote_plus(String, []).

quote_plus([], Acc) ->
    lists:reverse(Acc);

quote_plus([C | Rest], Acc) when ?QS_SAFE(C) ->
    quote_plus(Rest, [C | Acc]);

quote_plus([$\s | Rest], Acc) ->
    quote_plus(Rest, [$+ | Acc]);

quote_plus([C | Rest], Acc) ->
    <<Hi:4, Lo:4>> = <<C>>,
    quote_plus(Rest, [hexdigit(Lo), hexdigit(Hi), ?PERCENT | Acc]).

revjoin([], _Separator, Acc) ->
    Acc;

revjoin([S | Rest],Separator,[]) ->
    revjoin(Rest,Separator,[S]);

revjoin([S | Rest],Separator,Acc) ->
    revjoin(Rest,Separator,[S,Separator | Acc]).

hexdigit(C) when C < 10 -> $0 + C;
hexdigit(C) when C < 16 -> $A + (C - 10).

Answer 2

To answer my own question...I found this lib in ibrowse!

http://www.erlware.org/lib/5.6.3/ibrowse-1.4/ibrowse_lib.html#url_encode-1

url_encode/1

url_encode(Str) -> UrlEncodedStr

Str = string()
UrlEncodedStr = string()

URL-encodes a string based on RFC 1738. Returns a flat list.

I guess I can use this to do the encoding and still use http:

Answer 3

You can find here the YAWS urlencode and urldecode routines

They are fairly straightforward, although comments indicate the encode is not 100% complete for all punctuation characters.

Answer 4

I encountered the lack of this feature in the HTTP modules as well.

It turns out that this functionality is actually available in the erlang distribution, you just gotta look hard enough.

edoc_lib:escape_uri("look here i am") -> "look%20here%20i%20am"

edoc_lib

Answer 5

Hi, I’m trying to use this function. Actually it works in my local machine, either via erl shell either in a ejabberd module. But if I try to use this ejabberd module in production, when it gets to the point where it has to do edoc_lib:escape_uri the result is this :

E(:ejabberd_hooks:190) : {undef, [{edoc_lib,escape_uri, ["something to be escaped"]}, {mod_test,save_message,1}, {mod_test,on_message_sent,3}, {ejabberd_hooks,run1,3}, {ejabberd_c2s,session_established2,2}, {gen_fsm,handle_msg,7}, {proc_lib,wake_up,3}]}

Seems like the function is undefined. You know these erlang errors are not so explicit… Do I need to include something in order to have it work?

What I don’t understand is why the same function edoc_lib:escape_uri works in production if called inside an erlang shell…

EDIT : I answer myself, edoc_lib.beam was not in the path...

Answer 6

Here's a simple function that does the job. It's designed to work directly with inets httpc.

%% @doc A function to URL encode form data.
%% @spec url_encode(formdata()).

-spec(url_encode(formdata()) -> string()).
url_encode(Data) ->
    url_encode(Data,"").

url_encode([],Acc) ->
    Acc;

url_encode([{Key,Value}|R],"") ->
    url_encode(R, edoc_lib:escape_uri(Key) ++ "=" ++ edoc_lib:escape_uri(Value));
url_encode([{Key,Value}|R],Acc) ->
    url_encode(R, Acc ++ "&" ++ edoc_lib:escape_uri(Key) ++ "=" ++ edoc_lib:escape_uri(Value)).

Example usage:

    http:request(post, {"http://localhost:3000/foo", [], 
                        "application/x-www-form-urlencoded",
                        url_encode([{"username", "bob"}, {"password", "123456"}])}
                 ,[],[]).

Answer 7

Here's a "fork" of the edoc_lib:escape_uri function that improves on the UTF-8 support and also supports binaries.

escape_uri(S) when is_list(S) ->
    escape_uri(unicode:characters_to_binary(S));
escape_uri(<<C:8, Cs/binary>>) when C >= $a, C =< $z ->
    [C] ++ escape_uri(Cs);
escape_uri(<<C:8, Cs/binary>>) when C >= $A, C =< $Z ->
    [C] ++ escape_uri(Cs);
escape_uri(<<C:8, Cs/binary>>) when C >= $0, C =< $9 ->
    [C] ++ escape_uri(Cs);
escape_uri(<<C:8, Cs/binary>>) when C == $. ->
    [C] ++ escape_uri(Cs);
escape_uri(<<C:8, Cs/binary>>) when C == $- ->
    [C] ++ escape_uri(Cs);
escape_uri(<<C:8, Cs/binary>>) when C == $_ ->
    [C] ++ escape_uri(Cs);
escape_uri(<<C:8, Cs/binary>>) ->
    escape_byte(C) ++ escape_uri(Cs);
escape_uri(<<>>) ->
    "".

escape_byte(C) ->
    "%" ++ hex_octet(C).

hex_octet(N) when N =< 9 ->
    [$0 + N];
hex_octet(N) when N > 15 ->
    hex_octet(N bsr 4) ++ hex_octet(N band 15);
hex_octet(N) ->
    [N - 10 + $a].

Note that, because of the use of unicode:characters_to_binary it'll only work in R13 or newer.