views:

10

answers:

0

I was wondering if I could tokenize my string (lpcwszBuffer) using boost::tokenizer without making a copy of the string. I heard about using intrusive containers to avoid giant memory footprints, but I'm not really sure if it's applicable here.

If I was unclear, here's what I mean:

size_t Tokenize(const wchar_t* lpcwszBuffer, boost::scoped_array<wchar_t>* lppwszTokens)
{
   size_t nTokens = 0;

   if (lpcwszBuffer == NULL)
      return nTokens;

   boost::char_separator<wchar_t> Delimiters(L" ");

   //undesired
   //can I use an intrusive container here to increase speed?
   //afaik, the std::wstring class makes copies of the text buffer, but the buffer might be huge
   std::wstring wsBuffer = lpcwszBuffer;
   boost::tokenizer<boost::char_separator<wchar_t>, std::wstring::const_iterator, std::wstring> Tokens(wsBuffer, Delimiters);

   BOOST_FOREACH(std::wstring Token, Tokens)
   {
      StringCchCopy(lppwszTokens[nTokens++].get(), MAX_TOKEN_LEN, Token.c_str());
   }

   return nTokens;
}

Please ignore the mess; I had to remove some typedefs + get rid of other (nonrelated) code.