No, don't use an ArrayList of char values. That will box every char - performance will be horrible, as will memory usage. (Size of a reference + size of a boxed char for each character... yikes!)
Use a char[] internally and "resize" it (create a new array and copy the contents in) when you need to, perhaps doubling in size each time. (EDIT: You don't resize it to the exact size you need - you would start off with, say, 16 chars and keep doubling - so most Append operations don't need to "resize" the array.)
That's similar to how StringBuilder works anyway. (It's even closer to how Java's StringBuilder works.)
I suggest you actually build your own StringBuilder type with the most important members. Unit test the heck out of it, and profile where appropriate.
Let me know if you want a short example.