How to implement a web client in bash

Question

Are you able to write a Bash script that will download an HTTP resource from the web?

The inputs are:

$hostname 
$port     
$path     
$output

You can't:

• use external commands other than telnet (no sed, no awk, no wget, ...)
• use other shells

You can:

• use /dev/tcp pseudo devices
• use telnet

Your MUST pass this test (you can change the input if you want, I don't receive anything if you visit me):

hostname=andreafrancia.it # use what you want instead of this 
port=80
path=/bash-http-contest.txt

./your-script "$hostname" "$port" "$path" output.actual
wget http://"$hostname:$port/$path" -O output.expected

diff --binary output.actual output.expected # this should return 0 i.e. they are equals.

You MUST explicitly state that your code snippet uses one of these licenses: GPL (v1,v2,v3), or Apache License or that is in public domain or use some other license that allow someone else reuse your script.

You MUST put your code in the stackoverflow page (no external links or attached file).

The first win. The date that counts is the last edit date.

Answer 1

Try something like this:

(exec 3<>/dev/tcp/$hostname/$port
 echo -e "GET $path HTTP/1.1\r\nConnection: close\r\n\r\n" >&3
 cat <&3) > $output

Updated for Mike Ottum's bug fix.

Answer 2

Thanks to DigitalRoss, Mike Ottum and the other contributors I created the following that does 99% of the works.

I used the parameter expansion to remove headers. The problem is the last newline character of the page. This depends of the usage of the $() construct and I think that this problem couldn't be solved.

function download() {
    local hostname="$1"
    local port="$2"
    local path="$3"

    raw_output="$(download_raw "$hostname" "$port" "$path")"

    # strip the headers
    echo -n "${raw_output#*$'\r\n\r\n'}"
}

function download_raw() {
    local hostname="$1"
    local port="$2"
    local path="$3"

    (exec 3<>/dev/tcp/$hostname/$port
     echo -en "GET $path HTTP/1.1\r\nConnection: close\r\nHost: $hostname\r\n\r\n" >&3
     cat <&3)
}

hostname=andreafrancia.it
port=80
path=/

download "$hostname" "$port" "$path" > output.txt
wget http://"$hostname:$port/$path" -O output.expected
diff --binary output.txt output.expected

The result is:

[root@localhost ~]# diff --binary output.txt output.expected
74c74,75
< </html>
\ No newline at end of file
---
> </html>
>

Feel free to reuse and improve this solution.