I am using something similar to the following simplified script to parse snippets of python from a larger file:
import io
import tokenize
src = 'foo="bar"'
src = bytes(src.encode())
src = io.BytesIO(src)
src = list(tokenize.tokenize(src.readline))
for tok in src:
print(tok)
src = tokenize.untokenize(src)
Although the code is not the same in python2.x, it uses the same idiom and works just fine. However, running the above snippet using python3.0, I get this output:
(57, 'utf-8', (0, 0), (0, 0), '')
(1, 'foo', (1, 0), (1, 3), 'foo="bar"')
(53, '=', (1, 3), (1, 4), 'foo="bar"')
(3, '"bar"', (1, 4), (1, 9), 'foo="bar"')
(0, '', (2, 0), (2, 0), '')
Traceback (most recent call last):
File "q.py", line 13, in <module>
src = tokenize.untokenize(src)
File "/usr/local/lib/python3.0/tokenize.py", line 236, in untokenize
out = ut.untokenize(iterable)
File "/usr/local/lib/python3.0/tokenize.py", line 165, in untokenize
self.add_whitespace(start)
File "/usr/local/lib/python3.0/tokenize.py", line 151, in add_whitespace
assert row <= self.prev_row
AssertionError
I have searched for references to this error and its causes, but have been unable to find any. What am I doing wrong and how can I correct it?
[edit]
After partisann's observation that appending a newline to the source causes the error to go away, I started messing with the list I was untokenizing. It seems that the EOF
token causes an error if not immediately preceded by a newline so removing it gets rid of the error. The following script runs without error:
import io
import tokenize
src = 'foo="bar"'
src = bytes(src.encode())
src = io.BytesIO(src)
src = list(tokenize.tokenize(src.readline))
for tok in src:
print(tok)
src = tokenize.untokenize(src[:-1])