Problem with using getline and strtok together in a program .

Question

In the below program , I intend to read each line in a file into a string , break down the string and display the individual words.The problem I am facing is , the program now outputs only the first line in the file. I do not understand why this is happening ?

#include<iostream>
#include<string>
#include<fstream>
#include<cstdio>
using namespace std;

int main()
{
    ifstream InputFile("hello.txt") ;
    string store ;
    char * token;

    while(getline(InputFile,store))
    {
        cout<<as<<endl;
        token = strtok(&store[0]," ");
        cout<<token;
        while(token!=NULL)
        {
        token = strtok(NULL," ");
        cout<<token<<" ";
        }

    }

}

Answer 1

Well, there is a problem here. strtok() takes a null-terminated string, and the contents of a std::string are not necessarily null-terminated.

You can get a null-terminated string from a std::string by calling c_str() on it, but this returns a const char* (i.e., the string is not modifiable). strtok() takes a char* and modifies the string when it is called.

If you really want to use strtok(), then in my opinion the cleanest option would be to copy the characters from the std::string into a std::vector and the null-terminate the vector:

std::string s("hello, world");
std::vector<char> v(s.begin(), s.end());
v.push_back('\0');

You can now use the contents of the vector as a null-terminated string (using &v[0]) and pass that to strtok().

If you can use Boost, I'd recommend using Boost Tokenizer. It provides a very clean interface for tokenizing a string.

Answer 2

What James McNellis says is correct.

For a quick solution (though not the best), instead of

string store

use

const int MAX_SIZE_LINE = 1024; //or whatever value you consider safest in your context.
char store[MAX_SIZE_LINE];

Answer 3

I'm new to C++, but I think an alternative approach could be:

while(getline(InputFile, store))
{
    stringstream line(store); // include <sstream>
    string token;        

    while (line >> token)
    {
        cout << "Token: " << token << endl;
    }
}

This will parse your file line-by-line and tokenise each line based on whitespace separation (so this includes more than just spaces, such as tabs, and new lines).