It wasn't clear from your question - are you using a JTextArea or a TextArea? The reason it's ambiguous is people generally don't mix and match the light and heavy-weight frameworks (e.g. put awt components within a swing component).
If it's a JTextArea, I think your best bet is probably to use a DocumentListener.
DocumentListener myListener = ??;
JTextArea myArea = ??;
myArea.getDocument().addDocumentListener(myListener);
http://java.sun.com/docs/books/tutorial/uiswing/components/generaltext.html#doclisteners
I think what you'll need to do is listen for changes, and whenever something is added to the file, call the getLineCount() method, and compare it with the getRows() method. If it's broken the threshold, then use a setRows() call to increase the number of rows.
Will probably need to file some sort of UI change, especially to propagate up to the JTable.