How does one programmatically download files from the web?

Question

How are files downloaded from servers in programming languages like C? I understand higher level languages have magic functions like "download_file_from_url()" but they don't help me understand what is actually going on. I'm a little familiar with sockets but network programming in general is still a black box to me. Thanks for any help.

Answer 1

You should check out libcurl - it's open source so you can dig through it and see how a respected library approaches the problem.

Answer 2

Use a library like libcurl.

Answer 3

And a "black box" is probably a good way to keep it :-)

You do the same thing in C that you would do in "higher level languages" - use a library function that does it for you. (The difference is that the library function isn't a standard built-in part of the language).

One choice for C is libcurl

Answer 4

If you are downloading a file using HTTP then you should read RFC on HTTP (how data is split by chunks etc.), using FTP — RFC on FTP (which commands are used, e. g. PWD, CD etc.). However these are higher-level protocols that utilize sockets anyway.

Answer 5

To download a file (assume a simple case - no firewall etc...), you need to:

• Connect to a DNS server to resolve the name of the URL's server into an IP

• Open a connection to that IP on the URL's port or default port for your protocol (80 for http)

• Send the appropriate HTTP command over to that server

• Listen for HTTP response

• Process response correctly, and if the response contains the data for the file, keepr eding the reponse and saving the data in temp file

• When file is fully downloaded, close the connection and move the complete temp file into proper location.