The book publisher Manning offers a free download of the first edition of "Swing" by Matthew Robinson and Pavel Vorobiev at http://www.manning.com/robinson2. (Scroll down the page looking for the link "Download Complete Swing Book (MS Word 97).")
Chapter 20 talks about developing a WYSIWYG RTF editor using a JTextPane
as part of the editing component. The new version of the book is revised and describes the creation of a WYSIWYG HTML editor, but it is not free. (Despite what the page at the link says, the paper copy of the new edition does not appear to be available, but the eBook is, if you're interested.)
This was a great resource for me when I was trying to do something similar.