Usually they will overlay the textarea with their own display component like a div. As the user types, javascript will take the content typed and apply the styles in the display area. The value of the textarea is the text with the html tags needed to render it in the specified style. So visibly the user sees the styled text.
This is a simplified explanation of course.
I believe tinymce specifically uses a table/iframe for display purposes (which is substituted in place of the existing textarea). Once you're ready to save it copies all the info back to the textarea for processing.
Owen has the right idea. Those libraries replace the textarea with an iframe and then put the iframe's document into designMode or contentEditable mode. This literally enables you edit the html document in the iframe while the browser then handles the cursor and all keystrokes for you and gives you an api that can be called to make styling changes (bold, italic, and so on). Then when the user submits the form they copy the innerHTML from the iframe into the original textarea. For more details about how to enable this mode and what you can do with it see: https://developer.mozilla.org/En/Rich-Text_Editing_in_Mozilla
However, my suggestion to you is to use one of the existing rich text control libraries if you would like this ability on your site. I have built one before and found that you will spend huge amounts of time dealing with browser inconsistencies in order to get something that works well. Beyond the differences in how you enable editing features, you will also want to normalize the html that is created. For example, IE creates <br> elements when the user presses enter and FF creates <p> tags. For style changes IE uses <b>, <i>, etc. while FF uses spans with style attributes.