help with handling tokenization errors

Question

Hi,

The code is in python.

Please find below the piece of code that I use to tokenize a string.

strList = list(token[STRING] for token in generate_tokens(StringIO(line).readline) if token[STRING])

I get an error that reads like:-

    raise TokenError, ("EOF in multi-line statement", (lnum, 0))
tokenize.TokenError: ('EOF in multi-line statement', (2, 0))

I wish to ignore such errors and be able to complete the tokenization process. I have a lot of data, so I am okay with loosing a part of the data to these errors. However, I am not sure how to write the piece of code that would enable be to implement the desired functionality. Could some one help me out with the code please?

Thank you.

Edit1:-

on trying the

except tokenize.TokenError:
    pass

I get the following error message

    except tokenize.TokenError:
 NameError: name 'tokenize' is not defined

Answer 1

Notice that your error message says tokenize.TokenError. That is the type of Exception your code is raising. To catch the error, you use a try...except block. To skip the error you simply put pass in the except block.

import tokenize
try:
    strList = list(token[STRING] for token in tokenize.generate_tokens(StringIO(line).readline) if token[STRING])
except tokenize.TokenError:
    pass