views:

848

answers:

10

When doing a simple performance measurement, I was astonished to see that calling String.IndexOf(char) was actually slower than doing it manually! Is this really true?! Here is my test code:

const string str = @"91023m lkajsdfl;jkasdf;piou-09324\\adf \asdf\45\ 65u\ 86\ 8\\\;";
static int testIndexOf() { return str.IndexOf('\\'); }
static int testManualIndexOf() {
    string s = str;
    for (int i = 0; i < s.Length; ++i)
        if (s[i] == '\\') return i;
    return -1;
}

I ran each method 25 million times, and measured the time using a Stopwatch. The manual version was consistently 25% faster than the other one.

Any thoughts?!


Edit: Today I wanted to check this issue in a more comprehensive way, so I created a new project, and wrote the code you'll find below. I got the following results when running release mode from cmd for 100 million iterations:

  -   String.      IndexOf : 00:00:07.6490042
  - MyString.PublicIndexOf : 00:00:05.6676471
  - MyString.InternIndexOf : 00:00:06.1191796
  - MyString.PublicIndexOf2: 00:00:09.1363687
  - MyString.InternIndexOf2: 00:00:09.1182569

I ran these tests at least 20 times, getting almost the same results every time. My machine is XP SP3, VS 2008 SP1, P4 3.0 GHz with no hyper-threading and 1 GB RAM. I find the results really strange. As you can see, String.IndexOf was about 33% slower than my PublicIndexOf. Even stranger, I wrote the same method as internal, and it was about 8% slower than the public one! I do not understand what's happening, and I hope you could help me understand!

The testing code is below. (Sorry for the repeated code, but I found that using a delegate showed different timings, with the public and internal methods taking the same time.)

public static class MyString {
    public static int PublicIndexOf(string str, char value) {
        for (int i = 0; i < str.Length; ++i)
            if (str[i] == value) return i;
        return -1;
    }

    internal static int InternIndexOf(string str, char value) {
        for (int i = 0; i < str.Length; ++i)
            if (str[i] == value) return i;
        return -1;
    }

    public static int PublicIndexOf2(string str, char value, int startIndex) {
        if (startIndex < 0 || startIndex >= str.Length)
            throw new ArgumentOutOfRangeException("startIndex");
        for (; startIndex < str.Length; ++startIndex)
            if (str[startIndex] == value) return startIndex;
        return -1;
    }

    internal static int InternIndexOf2(string str, char value, int startIndex) {
        if (startIndex < 0 || startIndex >= str.Length)
            throw new ArgumentOutOfRangeException("startIndex");
        for (; startIndex < str.Length; ++startIndex)
            if (str[startIndex] == value) return startIndex;
        return -1;
    }
}

class Program {
    static void Main(string[] args) {
        int iterations = 100 * 1000 * 1000; // 100 millions
        char separator = '\\';
        string str = @"91023m lkajsdfl;jkasdf;piou-09324\\adf \asdf\45\ 65u\ 86\ 8\\\;";
        Stopwatch watch = new Stopwatch();

        // test String.IndexOf
        int sum = 0;
        watch.Start();
        for (int i = 0; i < iterations; ++i)
            sum += str.IndexOf(separator);
        watch.Stop();
        Console.WriteLine("  String.      IndexOf : ({0}, {1})", watch.Elapsed, sum);

        // test MyString.PublicIndexOf
        sum = 0;
        watch.Reset(); watch.Start();
        for (int i = 0; i < iterations; ++i)
            sum += MyString.PublicIndexOf(str, separator);
        watch.Stop();
        Console.WriteLine("MyString.PublicIndexOf : ({0}, {1})", watch.Elapsed, sum);

        // test MyString.InternIndexOf
        sum = 0;
        watch.Reset(); watch.Start();
        for (int i = 0; i < iterations; ++i)
            sum += MyString.InternIndexOf(str, separator);
        watch.Stop();
        Console.WriteLine("MyString.InternIndexOf : ({0}, {1})", watch.Elapsed, sum);

        // test MyString.PublicIndexOf2
        sum = 0;
        watch.Reset(); watch.Start();
        for (int i = 0; i < iterations; ++i)
            sum += MyString.PublicIndexOf2(str, separator,0);
        watch.Stop();
        Console.WriteLine("MyString.PublicIndexOf2: ({0}, {1})", watch.Elapsed, sum);

        // test MyString.InternIndexOf2
        sum = 0;
        watch.Reset(); watch.Start();
        for (int i = 0; i < iterations; ++i)
            sum += MyString.InternIndexOf2(str, separator,0);
        watch.Stop();
        Console.WriteLine("MyString.InternIndexOf2: ({0}, {1})", watch.Elapsed, sum);
    }
}
+2  A: 

Isn't it possible to download the code of the .net framework and to see what str.IndexOf really does?

tuinstoel
IndexOf is as extern method - not written in IL: [MethodImpl(MethodImplOptions.InternalCall)]public extern int IndexOf(char value, int startIndex, int count);
Bartek Szabat
Bartek is right. I tried opening it in Reflector and it showed the same signature.
Hosam Aly
+2  A: 

Is the time for this specific search actually critical, or are you doing the lethal "optimization without cause"? Did you try it with shorter and longer strings? With different search strings? The built-in function may be optimized for other cases where searching might be slower and your specific case ends up being slower. I wouldn't condemn the IndexOf API as being "always slower" until I did a bit more testing of a whole lot more cases.

ctacke
"We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil."---Donald Knuth
Jason
@Jason: Unthinkingly parroting this quote is largely why we have machines today that are thousands of times faster than those of 10 years ago, which take longer to do equivalent tasks - a whole generation of programmers who give no thought at all to efficiency.
Software Monkey
@Software Monkey: Unless the app that the OP is building is spending a large fraction of its time finding chars in strings, any perceived difference is irrelevant, and thus optimizing around it is premature.
Jason
Actually the whole application is about parsing financial stock data in real time, so yes, string searching is very critical. I guess this falls into the 3% category. :)
Hosam Aly
I agree that the API may be optimized for other cases, and that's why I am asking: because I don't have enough experience. Thanks for your help. :)
Hosam Aly
+8  A: 

Even if it is faster doing it "manually" it's not wise to do it. It's faster today, maybe, in your version of .net, your CLR setup. Maybe. At least for that particular string. But you can be pretty sure that the builtin will have better chances of improving with time. Go with your language. And IndexOf is much clearer too.

PEZ
Actually the question is not about whether to do it manually as much as it is about why the framework has a method that is slower than my first-look implementation! (Or, about what's wrong with my way of testing. :) )
Hosam Aly
Yes, I understood that you were not suggesting rolling your own indexOf. And I agree that your finding is interesting. Hopefully the framework maintainers will find this posting too and start working on it. =)
PEZ
+6  A: 

Perhaps these 25% is the cost of calling function.

Your method (testManualIndexOf) does not call any function.

Bartek Szabat
that would be easy to test, just change your manual to return myIndexOf(str); and put the for loop in the myIndexOf() method.
Jorn
I guessed it could be the cost of calling function, so I'll check it again when I return to work. But 25%? This looks much for a few push/pop instructions! I'll check it anyway.
Hosam Aly
I updated the testing code and the results are still the same. I updated the question with the current testing code.
Hosam Aly
+2  A: 

Well, you have an extra function call overhead in the first instance. Did you try wrapping your manual search in a method and calling it from your test -- that's really the best comparison since it's unlikely that you would want to write that loop every time you wanted to search for a character in a string.

tvanfosson
It seemed to me that the call overhead should have been optimized away by the JIT, but I'll check it anyway.
Hosam Aly
+6  A: 

Your test isn't really a fair one, as you aren't implementing the same functionality:

  1. You aren't passing any arguments to your manual test, instead using a const reference, which requires fewer op-codes to be executed than when arguments are passed as in the real IndexOf method.
  2. Your IndexOf test actually makes two method calls rather than one. It's likely the outer method call will be inlined in a Release build not run under the debugger, but your test doesn't indicate the conditions under which it was run (if this is the case, then you can ignore this point).

But essentially, given that your method does less, I'm not surprised it runs faster.

Greg Beech
1. I didn't think that the const reference would make much difference. After all, that's 25%! 2. I tested it in Release mode running from the command prompt, so I certainly hope that the JIT inlined the method.
Hosam Aly
+1  A: 

As was already said, I think the function call may account for some of the difference. Parameter passing and checking takes time.

I tried to see the code for IndexOf with Reflector but... bad luck, it's an internal CLR call. I think that alone should be enough to pick it over the manual approach, since it may improve with CLR improvements.

Martinho Fernandes
+5  A: 

There is definely sth wrong with your test or with your environment.

I just run this code:

    static void Main(string[] args)
    {
        {
            Stopwatch sw = new Stopwatch();
            sw.Start();
            for (int i = 0; i < 25000000; i++)
                testIndexOf();
            sw.Stop();
            Console.WriteLine(sw.Elapsed); //2.27 s
        }
        {
            Stopwatch sw = new Stopwatch();
            sw.Start();
            for (int i = 0; i < 25000000; i++)
                testManualIndexOf();
            sw.Stop();
            Console.WriteLine(sw.Elapsed); //9.81 s
        }

        Console.ReadLine();
    }

on two different machines, one Vista, one Linux/mono and found that internal IndexOf is 4 times faster.

Bartek Szabat
Unless your total elapsed time is greater than several seconds, there is way too much chance of "interference" skewing the results; these tests measuring in milliseconds are largely meaningless.
Software Monkey
ops, typo :) thx
Bartek Szabat
I tried it on Windows XP SP3 with VS 2008 SP1 on a P4 3GHz (no HT) processor. TestIndexOf() took 1.95 seconds, while TestManualIndexOf() took 1.45 seconds or so...
Hosam Aly
+5  A: 

(updated)

With your newly posted rig, I get the numbers below, which are in line with what I would expect:

String.      IndexOf : (00:00:06.3134669, -994967296)
MyString.PublicIndexOf : (00:00:07.0769368, -994967296)
MyString.InternIndexOf : (00:00:08.3463652, -994967296)
MyString.PublicIndexOf2: (00:00:12.0049268, -994967296)
MyString.InternIndexOf2: (00:00:12.4344756, -994967296)


(original)

I cannot reproduce your numbers in release at the console. I did 25M iterations, with results:

  • 1556ms, (825000000) (testIndexOf)
  • 1798ms, (825000000) (testManualIndexOf)

So IndexOf is quicker. I suspect your test rig is not being run in release mode at the command line (you can't run performance tests with the debugger attached; even the IDE adds too much overhead).

Here is my rig:

static void Main()
{
    const int LOOP = 25000000;
    int chk1 = 0, chk2 = 0;
    Stopwatch watch1 = Stopwatch.StartNew();
    for (int i = 0; i < LOOP; i++)
    {
        chk1 += testIndexOf();
    }
    watch1.Stop();
    Stopwatch watch2 = Stopwatch.StartNew();
    for (int i = 0; i < LOOP; i++)
    {
        chk2 += testManualIndexOf();
    }
    watch2.Stop();

    Console.WriteLine("{0}ms, ({1})", watch1.ElapsedMilliseconds, chk1);
    Console.WriteLine("{0}ms, ({1})", watch2.ElapsedMilliseconds, chk2);

}
Marc Gravell
Is there an error in your pasted results? LOOP=25,000,000, but the output indicates the loops ran 825,000,000 times?
Software Monkey
No, that indicates the item was found at index 33, 25M times
Marc Gravell
I ran my test in Release mode from the command prompt (but the IDE was open, does this make a difference?), and it gave me different results. (See my comment on Bartek Szabat's answer for the timings and machine configuration.) I shall post my test code when I return to work.
Hosam Aly
And BTW, if I ran it from the IDE or in Debug mode, shouldn't my code be slower too? (I would think the CLR shouldn't be affected in that mode, but my code wouldn't be optimized. Is this incorrect?)
Hosam Aly
Yes, "Debug" mode might typically include fewer optimisations; actually, this is a separate flag, but is cleared/set (by default) as you would expect on the Debug/Release configurations.
Marc Gravell
But would the optimization flag affect accessing CLR classes/methods? Shouldn't these be optimized already?
Hosam Aly
Your results agree with mine that the internal method is slower. I wonder why that is? It's the same code! It's also strange that your results are different than mine, so I'm asking a friend of mine to measure it. Do you think HT or dual-core can have an effect?
Hosam Aly
My friend ran the same executable, and got similar results to mine: 9.6, 6.8, 7, and 8.6 respectively.
Hosam Aly
Without optimization, it is hard to predict. The pre-compiled framework code shouldn't get slower, but the mechanism for *calling* it might change. Interesting, though. I wouldn't *expect* HT etc to make a difference, but the CPU *type* might. Bigger differences might be the OS (esp. x64 vs x86)...
Marc Gravell
...or the *exact* runtime version and compiler version used. I'm using 3.5SP1 on x86, on an Intel.
Marc Gravell
I'm using the same, with an old 3 GHz P4, with XP SP3 (32-bit). Framework is 3.5 SP1, as installed by VS 2008 SP1.
Hosam Aly
+1  A: 

I turned your code into an actual method, and added a test to check for null. I then wrote a loop to generate ten million random strings of random length (1 <= length <= 250) and compared the time using the instance method String.IndexOf and the custom testManualIndexOf. String.IndexOf took 5159 milliseconds and testManualIndexOf took 4838 milliseconds for a difference of 6.2%. Following is the not-so-pretty code:

public static void Main(string[] args) {
    int nTrials = 10000000;
    Random rg = new Random(1);
    Stopwatch sw = new Stopwatch();
    for (int i = 0; i < nTrials; i++) {
        int len = 1 + rg.Next(250);
        char[] randomChar = new char[len];
        for(int j = 0; j < len; j++) {
            randomChar[j] = GetRandomLetterOrDigit(rg);
        }
        string s = new string(randomChar);
        char findChar = GetRandomLetterOrDigit(rg);
        sw.Start();
        int index = s.IndexOf(findChar);
        //int index = testManualIndexOf(s, findChar);
        sw.Stop();
    }
    Console.WriteLine(sw.ElapsedMilliseconds);
}

private static char GetRandomLetterOrDigit(Random rg) {
    char c;
    int rc = rg.Next(62);

    if (rc < 26) {
        c = (char)('A' + rc);
    }
    else if (rc < 52) {
        c = (char)('a' + rc - 26);
    }
    else {
        c = (char)('0' + rc - 52);
    }
    return c;
}

private static int testManualIndexOf(string s, char c) {
    if (s == null) {
        throw new ArgumentNullException("s");
    }
    for (int i = 0; i < s.Length; ++i) {
        if (s[i] == c) return i;
    }
    return -1;
}
Jason
I'd be willing to bet that the time spent generating random numbers and allocating/deallocating char arrays and strings is significantly larger than the actual time spent doing string searching.
Adam Rosenfield
I agree with Adam.
Software Monkey
@Adam: You're right. Silly oversight on my part. I have changed the code to avoid that.
Jason
I think checking for null is irrelevant when comparing with String.IndexOf, since it is an instance method, unless they are avoiding the case where `this == null`! Of course a static method would need such a check, but I was more interested in the actual operations String.IndexOf needs.
Hosam Aly