Others have given you good links to resources on exactly how HTTPS works. I'll address the two reasons why it is used:
1. Security
When using HTTPS, the traffic between your browser and the web server is encrypted. This prevents anyone who happens to have access to any of the many wires that your data will traverse as it crosses the Internet from looking at what you are sending the server, or what the server is sending you. This is why HTTPS is used for sending passwords and other login credentials. This is one reason why websites dealing with banking and other matters that require privacy use HTTPS. This is why you probably want to use HTTPS if you are reading your webmail from a public wi-fi connection.
2. Identity
When connecting via HTTPS, the web server provides to you a certificate. In addition to containing the public key needed to facilitate the encryption mentioned above, the certificate also attempts to prove the identity of the web server. This prevents anyone who happens to have access to any of the many wires that your data will traverse as it crosses the Internet from diverting your traffic to their server instead, and pretending to be the website you wanted to contact. All the encryption in the world doesn't help if you established that encrypted connection with a hacker.
To do this, certificates are "signed" by a Certificate Authority. Certificate Authorities aim to verify that the person they issue a certificate to is who they claim to be. That is, a CA will not issue a "bankofamerica.com" certificate to anyone other than Bank of America. Your browser comes with a pre-installed set of Certificate Authorities whose signatures it trusts. If the certificate the server gives you is not signed by one of these trusted CAs, the browser will warn you.
Note that failure of the identity step does not impede the security step. If an HTTPS server gives you a certificate that is not signed by a CA that your browser trusts, you can still establish an encrypted, secure connection with the server - you just can't be sure of who is actually running the server that you're talking to.