How to split a text file into words?

Question

Hello, I am working on a assignment where I am supposed to read a file and count the number of lines and at the same time count the words in it. I tried a combination of getline and strtok inside a while loop, which did not work.

file:example.txt (the file to be read).

Hi, hello what a pleasant surprise.
Welcome to this place.
May you have a pleasant stay here.
(3 lines, and some words).

Readfile.cpp

#include <iostream>
#include <fstream>
#include<string>
using namespace std;
int main()
{
  ifstream in("example.txt");
  int count = 0;

  if(!in)
  {
    cout << "Cannot open input file.\n";
    return 1;
  }

  char str[255];
  string tok;
  char * t2;

  while(in)
  {
    in.getline(str, 255);
    in>>tok;
    char *dup = strdup(tok.c_str());
    do 
    {
        t2 = strtok(dup," ");
    }while(t2 != NULL);
    cout<<t2<<endl;
    free (dup);
    count++;
  }
  in.close();
  cout<<count;
  return 0;
}

Answer 1

Try moving your cout<<t2<<end; statement into your while loop.

That should make your code basically functional.

You may want to see this similar post for other approaches.

Answer 2

There are examples like this posted all over the internet. Here is a count-words program I wrote back when I was in high school. Use it as a starting point. Other things I would like to point out are:

std::stringstream :you std::getline the entire line, then use std::stringstream to chop it up into smaller pieces and tokenise it. You can get the entire line using std::getline and inputting it into a std::string, which you can then pass to std::stringstream.

Once again, this is only an example and won't do exactly what you want it to do, you will need to modify it yourself to make it do what you want it to do!

#include <iostream>
#include <map>
#include <string>
#include <cmath>
#include <fstream>

// Global variables
        std::map<std::string, int> wordcount;
        unsigned int numcount;

void addEntry (std::string &entry) {
        wordcount[entry]++;
        numcount++;
        return;
}


void returnCount () {
        double percentage = numcount * 0.01;
        percentage = floor(percentage + 0.5f);

        std::map<std::string, int>::iterator Iter;

        for (Iter = wordcount.begin(); Iter != wordcount.end(); ++Iter) {
                if ((*Iter).second > percentage) {
                        std::cout << (*Iter).first << " used " << (*Iter).second << " times" << std::endl;
                }
        }

}

int main(int argc, char *argv[]) {
        if (argc != 2) {
                std::cerr << "Please call the program like follows: \n\t" << argv[0] 
                        << " <file name>" << std::endl;
                return 1;
        }

        std::string data;

        std::ifstream fileRead;
        fileRead.open(argv[1]);
        while (fileRead >> data) {
                addEntry(data);
        }
        std::cout << "Total words in this file: " << numcount << std::endl;
        std::cout << "Words that are 1% of the file: " << std::endl;
        returnCount();
}

Answer 3

Ok, Just got this right!! Just removed all unnecessary code.

int main()
{    
    ifstream in("example.txt");
    int LineCount = 0;
    char* str = new char[500];

    while(in)
    {
        LineCount++;
        in.getline(str, 255);
        char * tempPtr = strtok(str," ");
        while(tempPtr)
        {
            AddWord(tempPtr, LineCount);
            tempPtr = strtok(NULL," ,.");
        }
    }
    in.close();
    delete [] str;
    cout<<"Total No of lines:"<<LineCount<<endl;
    showData();

    return 0;
}

Btw the original problem statement was to create a index program that would accept a user file and create an line-index of all words.

Thanks for all the help. This site is really great. No forum is useful unless the users are helpful. Thanks a lot.

Answer 4

If you can use boost libraries, I would suggest to use boost::tokenizer :

The boost Tokenizer package provides a flexible and easy to use way to break of a string or other character sequence into a series of tokens. Below is a simple example that will break up a phrase into words.

// simple_example_1.cpp
#include<iostream>
#include<boost/tokenizer.hpp>
#include<string>

int main(){
   using namespace std;
   using namespace boost;
   string s = "This is,  a test";
   tokenizer<> tok(s);
   for(tokenizer<>::iterator beg=tok.begin();beg!=tok.end();++beg){
       cout << *beg << "\n";
   }
}

Answer 5

I have not tried compiling this, but here's an alternative that is nearly as simple as using Boost, but without the extra dependency.

#include <iostream>
#include <sstream>
#include <string>

int main() {
  std::string line;
  while (std::getline(std::cin, line)) {
    std::istringstream linestream(line);
    std::string word;
    while (linestream >> word) {
      std::cout << word << "\n";
    }
  }
  return 0;
 }