Python regex string to list of words (including words with hyphens)

Question

I would like to parse a string to obtain a list including all words (hyphenated words, too). Current code is:

s = '-this is. A - sentence;one-word'
re.compile("\W+",re.UNICODE).split(s)

returns:

['', 'this', 'is', 'A', 'sentence', 'one', 'word']

and I would like it to return:

['', 'this', 'is', 'A', 'sentence', 'one-word']

Answer 1

You could use "[^\w-]+" instead.

Answer 2

Yo can try with the NLTK library:

>>> import nltk
>>> s = '-this is a - sentence;one-word'
>>> hyphen = r'(\w+\-\s?\w+)'
>>> wordr = r'(\w+)'
>>> r = "|".join([ hyphen, wordr])
>>> tokens = nltk.tokenize.regexp_tokenize(s,r)
>>> print tokens
['this', 'is', 'a', 'sentence', 'one-word']

I found it here: http://www.cs.oberlin.edu/~jdonalds/333/lecture03.html Hope it helps

Answer 3

If you don't need the leading empty string, you could use the pattern \w(?:[-\w]*\w)? for matching:

>>> import re
>>> s = '-this is. A - sentence;one-word'
>>> rx = re.compile(r'\w(?:[-\w]*\w)?')
>>> rx.findall(s)
['this', 'is', 'A', 'sentence', 'one-word']

Note that it won't match words with apostrophes like won't.

Answer 4

s = "-this is. A - sentence;one-word what's"
re.findall("\w+-\w+|[\w']+",s)

result: ['this', 'is', 'A', 'sentence', 'one-word', "what's"]

make sure you notice that the correct ordering is to look for hyypenated words first!

Answer 5

Here my traditional "why to use regexp language when you can use Python" alternative:

import string
s = "-this is. A - sentence;one-word what's"
s = filter(None,[word.strip(string.punctuation)
                 for word in s.replace(';','; ').split()
                 ])
print s
""" Output:
['this', 'is', 'A', 'sentence', 'one-word', "what's"]
"""