.NET Regular expressions on bytes instead of chars

Question

Hi,

I'm trying to do some parsing that will be easier using regular expressions.

The input is an array (or enumeration) of bytes.

I don't want to convert the bytes to chars for the following reasons:

1. Computation efficiency
2. Memory consumption efficiency
3. Some non-printable bytes might be complex to convert to chars. Not all the bytes are printable.

So I can't use Regex.

The only solution I know, is using Boost.Regex (which works on bytes - C chars), but this is a C++ library that wrapping using C++/CLI will take considerable work.

How can I use regular expressions on bytes in .NET directly, without working with .NET strings and chars?

Thank you.

Answer 1

Well, if I faced this problem, I would DO the C++/CLI wrapper, except I'd create specialized code for what I want to achieve. Eventually develop the wrapper with time to do general things, but this just an option.

The first step is to wrap the Boost::Regex input and output only. Create specialized functions in C++ that do all the stuff you want and use CLI just to pass the input data to the C++ code and then fetch the result back with the CLI. This doesn't look to me like too much work to do.

Update:

Let me try to clarify my point. Even though I may be wrong, I believe you wont be able to find any .NET Binary Regex implementation that you could use. That is why - whether you like it or not - you will be forced to choose between CLI wrapper and bytes-to-chars conversion to use .NET's Regex. In my opinion the wrapper is better choice, because it will be working faster. I did not do any benchmarking, this is just an assumption based on:

1. Using wrapper you just have to cast the pointer type (bytes <-> chars).
2. Using .NET's Regex you have to convert each byte of the input.

Answer 2

There is a bit of impedance mismatch going on here. You want to work with Regular expressions in .Net which use strings (multi-byte characters), but you want to work with single byte characters. You can't have both at the same time using .Net as per usual.

However, to break this mismatch down, you could deal with a string in a byte oriented fashion and mutate it. The mutated string can then act as a re-usable buffer. In this way you will not have to convert bytes to chars, or convert your input buffer to a string (as per your question).

An example:

//BLING
byte[] inputBuffer = { 66, 76, 73, 78, 71 };

string stringBuffer = new string('\0', 1000);

Regex regex = new Regex("ING", RegexOptions.Compiled);

unsafe
{
    fixed (char* charArray = stringBuffer)
    {
        byte* buffer = (byte*)(charArray);

        //Hard-coded example of string mutation, in practice you would
        //loop over your input buffers and regex\match so that the string
        //buffer is re-used.

        buffer[0] = inputBuffer[0];
        buffer[2] = inputBuffer[1];
        buffer[4] = inputBuffer[2];
        buffer[6] = inputBuffer[3];
        buffer[8] = inputBuffer[4];

        Console.WriteLine("Mutated string:'{0}'.",
             stringBuffer.Substring(0, inputBuffer.Length));

        Match match = regex.Match(stringBuffer, 0, inputBuffer.Length);

        Console.WriteLine("Position:{0} Length:{1}.", match.Index, match.Length);
    }
}

Using this technique you can allocate a string "buffer" which can be re-used as the input to Regex, but you can mutate it with your bytes each time. This avoids the overhead of converting\encoding your byte array into a new .Net string each time you want to do a match. This could prove to be very significant as I have seen many an algorithm in .Net try to go at a million miles an hour only to be brought to its knees by string generation and the subsequent heap spamming and time spent in GC.

Obviously this is unsafe code, but it is .Net.

The results of the Regex will generate strings though, so you have an issue here. I'm not sure if there is a way of using Regex that will not generate new strings. You can certainly get at the match index and length information but the string generation violates your requirements for memory efficiency.

Update

Actually after disassembling Regex\Match\Group\Capture, it looks like it only generates the captured string when you access the Value property, so you may at least not be generating strings if you only access index and length properties. However, you will be generating all the supporting Regex objects.