There are three approaches commonly taken in building a website-based chat system.
Java applet solution - either find or build a Java applet that communicates to a chat server. The applet can be an IRC client, or a custom-made chat client with a custom server. I've even seen some websites use a Java applet as a communication front-end, where the interface is all run by Javascript and HTML in the browser, but the Javascript sends and accepts events from the applet to run the chat.
AJAX Post/Poll - Every time a use writes a message, send the message to the HTTP server, where all the users connected are periodically polling for new messages.
Comet - Using mainly Javascript, each client establishes a long-term connection to an HTTP server, and idles. When a message is being sent from the user, it is send over the already pre-existing connection. And instead of polling for them, new messages from other users just flow down the same connection.
Personally, I find the 3rd option to be the most exciting, but the most complex as well. You will probably need to build your own version of an HTTP server to support the long-lived connection that Comet requires. And since there's a 16bit limit on the descriptors of sockets in TCP/IP, you'll be limited to about 64K sockets, per IP, on your server. (Remember, each client will need a open socket!) Finally, the techniques for building Comet client-side code are wildly different between browsers. There exist a few frameworks for that, but you'll have to maintain them while new browsers come out.
If you're running a small website, and you want to face a surmountable challenge, then just go with AJAX polling. It's fun, it's not too hard, and you'll learn a lot. If you can't be bothered, then just find a Java applet. Once it's configured with a matching server, you'll never have to worry about maintaining it, since that solution is very client-agnostic. Of course, it requires that the Java Runtime Environment be installed on the client, and that's not always going to be true...