In Java source code, double quote is a special character, used to declare string literals. You cannot have a double quote in a String literal without escaping it.
In general, I would try to avoid hard-coding strings like that in source code, particularly if I found myself doing it a lot - as you've noted, they're a pain to deal with as source and they might be quite likely to change, in which case you could do without recompiling. If you don't need to supply runtime parts to the text data, you could get away with something as simple as reading in the data from a properties file, or you could use a templating engine like Apache Velocity to keep the character data separate and still substitute variables at runtime - several of the examples in the linked user guide do exactly that with HTML.