You are first using () as delimiters, then switching to ,, but you are switching before extracting the first token (the text between parentheses).
What you probably intended to do is this:
transitionInput = data.nextLine();
st = new StringTokenizer(transitionInput,"()", false);
if (st.hasMoreTokens())
{
String chunk = st.nextToken();
st = new StringTokenizer(chunk, ",");
while (st.hasMoreTokens())
{
transition.add(st.nextToken());
}
}
This code assumes that the expression always starts and ends with parentheses. If this is the case, you may as well remove them manually using String.substring(). Also, you may want to consider using String.split() to do the actual splitting:
String transitionInput = data.nextLine();
transitionInput = transitionInput.substring(1, transitionInput.length() - 1);
for (String s : transitionInput.split(","))
transition.add(s);
Note that both examples assume that commas are used as separators, as in your sample code (although the text of your question says otherwise)