Python/YACC Lexer: Token priority?

Question

I'm trying to use reserved words in my grammar:

reserved = {
   'if' : 'IF',
   'then' : 'THEN',
   'else' : 'ELSE',
   'while' : 'WHILE',
}

tokens = [
 'DEPT_CODE',
 'COURSE_NUMBER',
 'OR_CONJ',
 'ID',
] + list(reserved.values())

t_DEPT_CODE = r'[A-Z]{2,}'
t_COURSE_NUMBER  = r'[0-9]{4}'
t_OR_CONJ = r'or'

t_ignore = ' \t'

def t_ID(t):
 r'[a-zA-Z_][a-zA-Z_0-9]*'
 if t.value in reserved.values():
  t.type = reserved[t.value]
  return t
 return None

However, the t_ID rule somehow swallows up DEPT_CODE and OR_CONJ. How can I get around this? I'd like those two to take higher precedence than the reserved words.

Answer 1

Two things spring to mind:

• obviously, the 'or' is a reserved word, like 'if', 'then' etc.
• your RE for t_ID matches a superset of the strings that are matched by DEPT_CODE.

Therefore I would solve it as follows: Include 'or' as reserved word and in t_ID, check if the length of the string is 2 and if it consists of uppercase letters only. If this is the case, return DEPT_CODE.