What is a good open source C word tokenizer library?
I am look for something like
Tokenize("there are three apples. One is orange, the other is blue,"
" and, finally, the last is yellow!")
with the output not containing any punctuation.
views:
142answers:
3On the given string - a string literal that is entitled to be placed in read-only memory - using `strtok()` is likely to give you a core dump or its moral equivalent.
Jonathan Leffler
2009-10-11 01:52:48
@Jonathan, if you have a string constant, strdup() it first. That's not a big task.
gnud
2009-10-11 16:09:04
A:
lex/flex is the classic tool, but it may be somewhat heavyweight for what you're doing.
Paul Nathan
2009-10-11 02:17:36
+1
A:
If the only need is to strip the punctuations, I'd use a for cycle that outputs (whatever it means in your context) the source string character by character, skipping the ispunct() ones.
ntd
2009-10-11 16:02:41