[C++] std::string manipulation: whitespace, "newline escapes '\'" and comments #

Question

Kind of looking for affirmation here. I have some hand-written code, which I'm not shy to say I'm proud of, which reads a file, removes leading whitespace, processes newline escapes '\' and removes comments starting with #. It also removes all empty lines (also whitespace-only ones). Any thoughts/recommendations? I could probably replace some std::cout's with std::runtime_errors... but that's not a priority here :)

const int RecipeReader::readRecipe()
{
    ifstream is_recipe(s_buffer.c_str());
    if (!is_recipe)
        cout << "unable to open file" << endl;
    while (getline(is_recipe, s_buffer))
    {
        // whitespace+comment
        removeLeadingWhitespace(s_buffer);
        processComment(s_buffer);
        // newline escapes + append all subsequent lines with '\'
        processNewlineEscapes(s_buffer, is_recipe);
        // store the real text line
        if (!s_buffer.empty())
            v_s_recipe.push_back(s_buffer);
        s_buffer.clear();
    }
    is_recipe.close();
    return 0;
}

void RecipeReader::processNewlineEscapes(string &s_string, ifstream &is_stream)
{
    string s_temp;
    size_t sz_index = s_string.find_first_of("\\");
    while (sz_index <= s_string.length())
    {
        if (getline(is_stream,s_temp))
        {
            removeLeadingWhitespace(s_temp);
            processComment(s_temp);
            s_string = s_string.substr(0,sz_index-1) + " " + s_temp;
        }
        else
            cout << "Error: newline escape '\' found at EOF" << endl;
        sz_index = s_string.find_first_of("\\");
    }
}

void RecipeReader::processComment(string &s_string)
{
    size_t sz_index = s_string.find_first_of("#");
    s_string = s_string.substr(0,sz_index);
}

void RecipeReader::removeLeadingWhitespace(string &s_string)
{
    const size_t sz_length = s_string.size();
    size_t sz_index = s_string.find_first_not_of(" \t");
    if (sz_index <= sz_length)
    s_string = s_string.substr(sz_index);
    else if ((sz_index > sz_length) && (sz_length != 0)) // "empty" lines with only whitespace
        s_string.clear();
}

Some extra info: the first s_buffer passed to the ifstream contains the filename, std::string s_buffer is a class data member, so is std::vector v_s_recipe. Any comment is welcome :)

UPDATE: for the sake of not being ungrateful, here is my replacement, all-in-one function that does what I want for now (future holds: parenthesis, maybe quotes...):

void readRecipe(const std::string &filename)
{
    string buffer;
    string line;
    size_t index;
    ifstream file(filename.c_str());
    if (!file)
        throw runtime_error("Unable to open file.");

    while (getline(file, line))
    {
        // whitespace removal
        line.erase(0, line.find_first_not_of(" \t\r\n\v\f"));
        // comment removal TODO: store these for later output
        index = line.find_first_of("#");
        if (index != string::npos)
            line.erase(index, string::npos);
        // ignore empty buffer
        if (line.empty())
            continue;
        // process newline escapes
        index = line.find_first_of("\\");
        if (index != string::npos)
        {
            line.erase(index,string::npos); // ignore everything after '\'
            buffer += line;
            continue; // read next line
        }
        else // no newline escapes found
        {
            buffer += line;
            recipe.push_back(buffer);
            buffer.clear();
        }
    }
}

Answer 1

Definitely ditch the hungarian notation.

Answer 2

It's not bad, but I think you're thinking of std::basic_string<T> too much as a string and not enough as an STL container. For example:

void RecipeReader::removeLeadingWhitespace(string &s_string)
{
    s_string.erase(s_string.begin(), 
        std::find_if(s_string.begin(), s_string.end(), std::not1(isspace)));
}

Answer 3

I'm not big on methods that modify the parameters. Why not return strings rather than modifying the input arguments? For example:

string RecipeReader::processComment(const string &s)
{
    size_t index = s.find_first_of("#");
    return s_string.substr(0, index);
}

I personally feel this clarifies intent and makes it more obvious what the method is doing.

Answer 4

I'd consider replacing all your processing code (almost everything you've written) with boost::regex code.

Answer 5

A few comments:

• As another answer (+1 from me) said - ditch the hungarian notation. It really doesn't do anything but add unimportant trash to every line. In addition, ifstream yielding an is_ prefix is ugly. is_ usually indicates a boolean.
• Naming a function with processXXX gives very very little information on what it is actually doing. Use removeXXX, like you did with the RemoveLeadingWhitespace function.
• The processComment function does an unnecessary copy and assignment. Use s.erase(index, string::npos); (npos is default, but this is more obvious).
• It's not clear what your program does, but you might consider storing a different file format (like html or xml) if you need to post-process your files like this. That would depend on the goal.
• using find_first_of('#') may give you some false positives. If it's present in quotes, it's not necessarily indicating a comment. (But again, this depends on your file format)
• using find_first_of(c) with one character can be simplified to find(c).
• The processNewlineEscapes function duplicates some functionality from the readRecipe function. You may consider refactoring to something like this:

-

string s_buffer;
string s_line;
while (getline(is_recipe, s_line)) {
  // Sanitize the raw line.
  removeLeadingWhitespace(s_line);
  removeComments(s_line);
  // Skip empty lines.
  if (s_line.empty()) continue;
  // Add the raw line to the buffer.
  s_buffer += s_line;
  // Collect buffer across all escaped lines.
  if (*s_line.rbegin() == '\\') continue;
  // This line is not escaped, now I can process the buffer.
  v_s_recipe.push_back(s_buffer);
  s_buffer.clear();
}

Answer 6

A few comments:

• If s_buffer contains the file name to be opened, it should have a better name like s_filename.
• The s_buffer member variable should not be reused to store temporary data from reading the file. A local variable in the function would do as well, no need for the buffer to be a member variable.

• If there is not need to have the filename stored as a member variable it could just be passed as a parameter to readRecipe()

• processNewlineEscapes() should check that the found backslash is at the end of the line before appending the next line. At the moment any backslash at any position triggers adding of the next line at the position of the backslash. Also, if there are several backslashes, find_last_of() would probably easier to use than find_first_of().

• When checking the result of find_first_of() in processNewlineEscapes() and removeLeadingWhitespace() it would be cleaner to compare against string::npos to check if anything was found.

• The logic at the end of removeLeadingWhitespace() could be simplified:

  size_t sz_index = s_string.find_first_not_of(" \t");
  if (sz_index != s_string.npos)
     s_string = s_string.substr(sz_index);
  else // "empty" lines with only whitespace
     s_string.clear();
  

Answer 7

You might wish to have a look at Boost.String. It's a simple collection of algorithms to work with streams, and notably features trim methods :)

Now, on to the review itself:

Don't bother to remove the hungarian notation, if it's your style then use it, however you should try and improve the names of methods and variables. processXXX is definitely not indicating anything useful...

Functionally, I am worried about your assumptions: the main issue here is that you do not care for espace sequences (\n uses a backslash for example) and you do not worry for the presence of strings of charachters: std::cout << "Process #" << pid << std::endl; would yield an invalid line because of your "comment" preprocessing

Furthermore, since you remove the comments before processing the newline escapes:

i = 3; # comment \
         running comment

will be parsed as

i = 3; running comment

which is syntactically incorrect.

From an interface point of view: there is not benefit in having the methods being class members here, you don't need an instance of RecipeReader really...

And finally, I find it awkward that two methods would read from the stream.

Little peeve of mine: returning by const value does not serve any purpose.

Here is my own version, as I believe than showing is easier than discussing:

// header file

std::vector<std::string> readRecipe(const std::string& fileName);

std::string extractLine(std::ifstream& file);

std::pair<std:string,bool> removeNewlineEscape(const std::string& line);
std::string removeComment(const std::string& line);

// source file

#include <boost/algorithm/string.hpp>

std::vector<std::string> readRecipe(const std::string& fileName)
{
  std::vector<std::string> result;

  ifstream file(fileName.c_str());
  if (!file) std::cout << "Could not open: " << fileName << std::endl;

  std::string line = extractLine(file);
  while(!line.empty())
  {
    result.push_back(line);
    line = extractLine(file);
  } // looping on the lines

  return result;
} // readRecipe


std::string extractLine(std::ifstream& file)
{
  std::string line, buffer;
  while(getline(file, buffer))
  {
    std::pair<std::string,bool> r = removeNewlineEscape(buffer);
    line += boost::trim_left_copy(r.first); // remove leading whitespace
                                            // based on the current locale
    if (!r.second) break;
    line += " "; // as we append, we insert a whitespace
                 // in order unintended token concatenation
  }

  return removeComment(line);
} // extractLine

//< Returns the line, minus the '\' character
//<         if it was the last significant one
//< Returns a boolean indicating whether or not the line continue
//<         (true if it's necessary to concatenate with the next line)
std::pair<std:string,bool> removeNewlineEscape(const std::string& line)
{
  std::pair<std::string,bool> result;
  result.second = false;

  size_t pos = line.find_last_not_of(" \t");
  if (std::string::npos != pos && line[pos] == '\')
  {
    result.second = true;
    --pos; // we don't want to have this '\' character in the string
  }

  result.first = line.substr(0, pos);
  return result;
} // checkNewlineEscape

//< The main difficulty here is NOT to confuse a # inside a string
//< with a # signalling a comment
//< assuming strings are contained within "", let's roll
std::string removeComment(const std::string& line)
{
  size_t pos = line.find_first_of("\"#");
  while(std::string::npos != pos)
  {
    if (line[pos] == '"')
    {
      // We have detected the beginning of a string, we move pos to its end
      // beware of the tricky presence of a '\' right before '"'...
      pos = line.find_first_of("\"", pos+1);
      while (std::string::npos != pos && line[pos-1] == '\')
        pos = line.find_first_of("\"", pos+1);
    }
    else // line[pos] == '#'
    {
      // We have found the comment marker in a significant position
      break;
    }
    pos = line.find_first_of("\"#", pos+1);
  } // looking for comment marker

  return line.substr(0, pos);
} // removeComment

It is fairly inefficient (but I trust the compiler for optmizations), but I believe it behaves correctly though it's untested so take it with a grain of salt. I have focused mainly on solving the functional issues, the naming convention I follow is different from yours but I don't think it should matter.