views:

746

answers:

5

Hello!

I would like to split a text into single words using PHP. Do you have any idea how to achieve this?

My approach:

function tokenizer($text) {
    $text = trim(strtolower($text));
    $punctuation = '/[^a-z0-9äöüß-]/';
    $result = preg_split($punctuation, $text, -1, PREG_SPLIT_NO_EMPTY);
    for ($i = 0; $i < count($result); $i++) {
        $result[$i] = trim($result[$i]);
    }
    return $result; // contains the single words
}
$text = 'This is an example text, it contains commas and full-stops. Exclamation marks, too! Question marks? All punctuation marks you know.';
print_r(tokenizer($text));

Is this a good approach? Do you have any idea for improvement?

Thanks in advance!

+2  A: 

Tokenize - strtok.

<?php
$text = 'This is an example text, it contains commas and full stops. Exclamation marks, too! Question marks? All punctuation marks you know.';
$delim = ' \n\t,.!?:;';

$tok = strtok($text, $delim);

while ($tok !== false) {
    echo "Word=$tok<br />";
    $tok = strtok($delim);
}
?>
eed3si9n
Thank you, I think this function does this well.
This won't work if you get a : or ; or any other punctuation character you haven't accounted for.
marcog
@marcog, I added : and ;. Doesn't {P} catch apostrophe and hyphen?
eed3si9n
What about cases such quoting? My updated answer discriminates between these cases.
marcog
A: 

Do:

str_word_count($text, 1);

Or if you need unicode support:

function str_word_count_Helper($string, $format = 0, $search = null)
{
    $result = array();
    $matches = array();

    if (preg_match_all('~[\p{L}\p{Mn}\p{Pd}\'\x{2019}' . preg_quote($search, '~') . ']+~u', $string, $matches) > 0)
    {
     $result = $matches[0];
    }

    if ($format == 0)
    {
     return count($result);
    }

    return $result;
}
Alix Axel
Thanks but this wouldn't work. "Fri3nd" wouldn't be extracted but it should.
I don't understand why "Fri3nd" should be extracted. Removed from the array, broken down into "Fri3" and "nd" (or similar)? O.o
David Thomas
If you want to consider numbers as words just do str_word_count_Helper($string, 1, '0123456789');
Alix Axel
+4  A: 

Use the class \p{P} which matches any unicode punctuation character, combined with the \s whitespace class.

$result = preg_split('/((^\p{P}+)|(\p{P}*\s+\p{P}*)|(\p{P}+$))/', $text, -1, PREG_SPLIT_NO_EMPTY);

This will split on a group of one or more whitespace characters, but also suck in any surrounding punctuation characters. It also matches punctuation characters at the beginning or end of the string. This discriminates cases such as "don't" and "he said 'ouch!'"

marcog
+1, not sure, tho, how this will deal with äöüß. Does regex normally classify äöüß as word characters?
Peter Perháč
Thank you. This would't probably work for English texts but I also want to extract German umlauts (ä, ö, ü), the "ß" and numbers in a string. The "\W" wouldn't extract "Fri3nd", would it?
Seems it does not, but updated answer with something similar that works.
marcog
Updated answer works with perl (which php regex are based on): $ echo "äöüß, test" | perl -e 'while (<>) { if (/([\p{P}\s]+)/) { print "$1\n"; } }',
marcog
+1 You hit it with the Unicode character properties!
Gumbo
Should one split don't into don and t?
eed3si9n
Updated it to handle such a case :)
marcog
Thanks, marcog, it works perfectly!But is it really better than my updated code above? Actually, what is the difference between our approaches? Is one faster than the other one?
In your approach you're specifying the non-punctuation characters. You will be therefore be missing some cases, e.g. á. Why try manually specify them when the whole set of unicode punctuation characters has already been defined? And like eed3si9n pointed out with my original answer, yours will break up words such as don't.
marcog
Convinced me! ;) Thanks!
A: 

you can also use PHP strtok() function to fetch string tokens from your large string. you can use it like this:

 $result = array();
 // your original string
 $text = 'This is an example text, it contains commas and full stops. Exclamation marks, too! Question marks? All punctuation marks you know.';
 // you pass strtok() your string, and a delimiter to specify how tokens are separated. words are seperated by a space.
 $word = strtok($text,' ');
 while ( $word !== false ) {
     $result[] = $word;
     $word = strtok(' ');
 }

see more on php documentation for strtok()

farzad
+1  A: 

I would first make the string to lower-case before splitting it up. That would make the i modifier and the array processing afterwards unnecessary. Additionally I would use the \W shorthand for non-word characters and add a + multiplier.

$text = 'This is an example text, it contains commas and full stops. Exclamation marks, too! Question marks? All punctuation marks you know.';
$result = preg_split('/\W+/', strtolower($text), -1, PREG_SPLIT_NO_EMPTY);


Edit   Use the Unicode character properties instead of \W as marcog suggested. Something like [\p{P}\p{Z}] (punctuation and separator characters) would cover the characters more specific than \W.

Gumbo
Thanks, the idea to perform strtolower() before is very good. I'll use this.