Tokenizing strings in c

Question

I have been trying to tokenize a string using SPACE as delimiter but it doesn't work. Does any one have suggestion on why it doesn't work?

Edit: tokenizing using:

strtok(string, " ");

the code is like the following

pch = strtok (str," ");
while (pch != NULL)
{
  printf ("%s\n",pch);
  pch = strtok (NULL, " ");
}

Answer 1

Add some code and let us see what you're doing.

Answer 2

Are you using strtok or something you grew yourself?

http://www.cplusplus.com/reference/clibrary/cstring/strtok.html

If you are using strtok are you trying to do it on a constant string?

Answer 3

When reading the strtok documentation, I see you need to pass in a NULL pointer after the first "initializing" call. Maybe you didn't do that. Just a guess of course.

Answer 4

do it like this:

char s[256];
strcpy(s, "one two three");
char* token = strtok(s, " ");
while (token) {
    printf("token: %s\n", token);
    token = strtok(NULL, " ");
}

Note: strtok modifies the string its tokenising, so it cannot be a const char*.

Answer 5

Here's an example of strtok usage, keep in mind that strtok is destructive of its input string (and therefore can't ever be used on a string constant

char *p = strtok(str, " ");
while(p != NULL) {
    printf("%s\n", p);
    p = strtok(NULL, " ");
}

Basically the thing to note is that passing a NULL as the first param to strtok tells it to get the next token from the string it was previously tokenizing.

Answer 6

You can simplify the code by introducing an extra variable.

#include <string.h>
#include <stdio.h>

int main()
{
    char str[100], *s = str, *t = NULL;

    strcpy(str, "a space delimited string");
    while ((t = strtok(s, " ")) != NULL) {
     s = NULL;
     printf(":%s:\n", t);
    }
    return 0;
}

Answer 7

strtok can be very dangerous. It is not thread safe. Its intended use is to be called over and over in a loop, passing in the output from the previous call. The strtok function has an internal variable that stores the state of the strtok call. This state is not unique to each thread - it is global. If any other code uses strtok in another thread, you get problems. Not the kind of problems you want to track down either!

I'd recommend looking for a regex implementation, or using sscanf to pull apart the string.

Try this:

char strprint[256];
char text[256];
strcpy(text, "My string to test");
while ( sscanf( text, "%s %s", strprint, text) > 0 ) {
   printf("token: %s\n", strprint);
}

Note: The 'text' string is destroyed as it's separated. This may not be the preferred behaviour =)