I was wondering if I could tokenize my string (lpcwszBuffer
) using boost::tokenizer
without making a copy of the string. I heard about using intrusive containers to avoid giant memory footprints, but I'm not really sure if it's applicable here.
If I was unclear, here's what I mean:
size_t Tokenize(const wchar_t* lpcwszBuffer, boost::scoped_array<wchar_t>* lppwszTokens)
{
size_t nTokens = 0;
if (lpcwszBuffer == NULL)
return nTokens;
boost::char_separator<wchar_t> Delimiters(L" ");
//undesired
//can I use an intrusive container here to increase speed?
//afaik, the std::wstring class makes copies of the text buffer, but the buffer might be huge
std::wstring wsBuffer = lpcwszBuffer;
boost::tokenizer<boost::char_separator<wchar_t>, std::wstring::const_iterator, std::wstring> Tokens(wsBuffer, Delimiters);
BOOST_FOREACH(std::wstring Token, Tokens)
{
StringCchCopy(lppwszTokens[nTokens++].get(), MAX_TOKEN_LEN, Token.c_str());
}
return nTokens;
}
Please ignore the mess; I had to remove some typedefs + get rid of other (nonrelated) code.