Python strings split with multiple separators

Question

Weird - I think what I want to do is a fairly common task but I've found no reference on the web. I have text, with punctuation, and I want an array of the words. i.e - "Hey, you - what are you doing here!?" should be ['hey', 'you', 'what', 'are', 'you', 'doing', 'here']. But python's split() only works with one argument... so I have all words with punctuation after I split with white space. Any ideas?

Thanks

Answer 1

A case where regular expressions are justified:

import re
DATA = "Hey, you - what are you doing here!?"
print re.findall(r'\w+', DATA)
# Prints ['Hey', 'you', 'what', 'are', 'you', 'doing', 'here']

Answer 2

You want Python's RegEx module's findall() method:

http://www.regular-expressions.info/python.html

Example

Answer 3

re.split()

re.split(pattern, string[, maxsplit=0])

Split string by the occurrences of pattern. If capturing parentheses are used in pattern, then the text of all groups in the pattern are also returned as part of the resulting list. If maxsplit is nonzero, at most maxsplit splits occur, and the remainder of the string is returned as the final element of the list. (Incompatibility note: in the original Python 1.5 release, maxsplit was ignored. This has been fixed in later releases.)

>>> re.split('\W+', 'Words, words, words.')
['Words', 'words', 'words', '']
>>> re.split('(\W+)', 'Words, words, words.')
['Words', ', ', 'words', ', ', 'words', '.', '']
>>> re.split('\W+', 'Words, words, words.', 1)
['Words', 'words, words.']

Answer 4

try this:

import re

phrase = "Hey, you - what are you doing here!?"
matches = re.findall('\w+', phrase)
print matches

this will print ['Hey', 'you', 'what', 'are', 'you', 'doing', 'here']

Answer 5

Another way to achieve this is to use the Natural Language Tool Kit (nltk).

import nltk
data= "Hey, you - what are you doing here!?"
word_tokens = nltk.tokenize.regexp_tokenize(data, r'\w+')
print word_tokens

This prints: ['Hey', 'you', 'what', 'are', 'you', 'doing', 'here']

The biggest drawback of this method is that you need to install the nltk package.

The benefits are that you can do a lot of fun stuff with the rest of the nltk package once you get your tokens.

Answer 6

Use list comprehensions for this stuff...it seems easier

data= "Hey, you - what are you doing here!?"
tokens = [c for c in data if c not in (',', ' ', '-', '!', '?')]

I find this easier to comprehend (read..maintain) than using regexp, simply because I am not that good at regexp...which is the case with most of us :) . Also if you know what set of separators you might be using, you can keep them in a set. With a very huge set, this might be slower...but the 're' module is slow as well.

Answer 7

another way, without regex

import string
punc = string.punctuation
thestring =  "Hey, you - what are you doing here!?"
s=list(thestring)
''.join([ o for o in s if not o in string.punctuation ]).split()

Answer 8

Kinda late answer :), but I had a similar dilemma and didn't want to use 're' module.

def my_split(s, seps):
    res = [s]
    for sep in seps:
        s, res = res, []
        for seq in s:
            res += seq.split(sep)
    return res

print my_split('1111  2222 3333;4444,5555;6666', [' ', ';', ','])
['1111', '', '2222', '3333', '4444', '5555', '6666']