What is the best way to compare .NET performance vs. VB 6 performance at a customer site?

Question

Two questions:

1. Can someone point me to unbiased data that compares .NET performance to VB 6 performance? I have searched but it is surprisingly difficult to find.
2. What is the best way to compare .NET performance to VB 6 performance as an app behaves at a customer's site?

We have a WindowsForms, client-server app (written for 2.0, upgrading to 3.5 SP 1 soon) about which certain customers complain of "slow performance" as compared to the previous VB 6 version. I know, "slow performance" is very vague and general, but is it true to assume .NET code might be slower than VB 6 code because .NET runs in a VM? I wrote 100% of the code in C#, so it was not ported by some third person or wizard.

Not all customers make this complaint, so we suspect something environmental. Is our only option to measure performance at a customer site? Some of our customers use SQL Server 2005 on Windows Server 2003 on a Novell network. Would they see dramatically different data access performance than a similar machine on a Windows network?

Answer 1

Performance pretty much always depends on what you're doing.

If possible, go on-site and watch the customer using the app. Find out where the slowdown is happening, then profile that - ideally with a similar amount of data etc.

Check the performance of both the client and the server at the customer site - find out which is actually causing the problem. Check the network configuration and health - silly settings can sometimes keep things working but painfully slow.

Answer 2

The performance versus the VB6 app is probably due more to application design than c# versus VB6. I would guestimate that 9.5/10 times the C# app would be faster if designed properly. .Net does not run inside of a VM. The CLR JIT compiles a .Net app to machine level code, making it pretty darn fast. The reason is runs a bit slower than unmanged code is that the sandbox .Net run under is managed, and that has some overhead.

Answer 3

1. I haven't ever looked for this kind of data. You might look at the canonical "Pet Shop" app that is often used for benchmarking purposes. It's probably available in both flavors of dev platform. I suspect #2 is far more important to answer though. Who cares if Pet Shop runs quickly. It's your customer's app that matters.

2. Most performance issues are database and/or network related. If you made significant database changes between versions, I'd profile those first using the SQL tracing/profiling tools or just running some simple tests on key procs/sql that execute to support your so called "slow performance" forms.

If I understand you correctly, it seems to be fine at some clients but not at others, and somebody is hammering you on the fact that it is .NET and VB6 was just fine, thank you very much. Network differences between clients is definitely a critical difference. Server OS, speed of switches and network infrastructure, etc. - so many variables to explore. But I'd be willing to bet it has nothing to do with .NET vs. VB6.

Answer 4

If you're talking about effectively the same client application written in VB 6.0 and VB.NET then the VB 6.0 app will most likely get to the first screen more quickly (because of .NET JIT compliation and asssembly loading). After that the performance should be about the same.

The problem is - in any real situation - the applications would be architeched differently and a true performance comparison would not be meaningful,

Answer 5

The first thing I need to mention is that .Net code never runs in a VM. While it's true .Net programs are compiled to IL that is somewhat analogous to Java's ByteCode, the important difference is that the IL is also in turn compiled to fully native code before the app starts, rather than interpreted by a VM.

Moving on to the .Net/VB6 comparison. I can't point to specific data, but from personal experience it depends on what you're doing.

To illustrate this, let's think about six different benchmark applications, each with a vb6 and .Net version. Each application picks out a specific operation, performs it 100,000 times, and then records the amount of time taken. Note that this is a thought experiment: I haven't actually seen results from real apps. But I feel like I have a sense of the strengths and weaknesses of both platforms.

• Application A does some serious cpu-heavy number crunching. In this case, I have to believe the results here would be almost identical. Both VB6 and .Net compile down to native code, and so a cpu mult instruction is the same regardless. That said, if you're not using Option Strict you could quickly get yourself in trouble on either platform. Since you used C# (which essentially is always Option Strict), that could give your .Net code an advantage.

• Application B goes out and retrieves a value from a database. Again, results are probably very close, though I'd have to give .Net a very slight edge for two reasons: it uses a native sql client, which is supposedly a little faster, and it does things like automatic connection pooling and makes it easier to factor out things like connecting once and re-using that connection rather than connecting repeatedly. But if you compare apples to apples as far as code goes, the two will likely be very close.

• Application C shows and hides a form repeatedly. Here I would have to give vb6 the nod. It's based on an older, less glitzy widget set which will frankly be cheaper to render. Also, the WinForms components aren't known for being all that speedy. However, the difference probably isn't as large as you might think, and you're probably not completely redrawing forms that often, either. This is likely the slowness that your users complain about.

• Application D is based on string operations, and here I have to give the nod to .Net. Both VB6 and .Net are known for slow string operations. However, .Net provides a number of tools that are simply not available in vb6 to help developers overcome the slowness. That said, if you're not aware of those tools, poor string operation choices could easily bog down your app doing useless work. This is also likely contributing to user complaints.

• Application E is going to repeatedly load an Xml document into memory. Again, I have to think that .Net would have an advantage. First of all the xml tools available to vb were primitive at best. I find it unlikely they haven't been improved significantly. Additionally, .Net's garbage collector is pretty smart, giving it an advantage or allocated and freeing memory during the loads. However, I think the biggest thing here is going to be disk speed, which means the difference wouldn't be that large.

• Application F will repeatedly start a .Net or vb6 program, wait for it to become ready (for some definition of "ready"), and then close it. Here vb6 likely has the advantage, for two reasons. Firstly, the .Net has to compile the IL to bytecode on the first run. Thankfully, that's only the first run. Seconly, the .Net app also has to load any referenced assemblies. By comparison, the VB6 runtime is much smaller. However, this is normally only true the first time you start the app on a particular machine, and there are ways to pre-compile the .Net code. This may also contribute to perceived slowness.

One important point is that for all of these applications, the .Net app is likely going to have a much larger memory footprint. For some reasons developers tend to think about this as a bad performance characteristic when in fact the opposite is true. If your app is using more memory, it's probably spending less time going to disk, and the disk is slower by far. It's probably also caching more, which saves cpu time. And for .Net specifically, it's saving up allocation/deallocation operations for periods when it's more important or when the pc is otherwise idle.

I don't have time, but it's be interesting to see someone use the StopWatch implementation (at least for the .Net side) mentioned in today's Coding Horror to get real benchmarks for each of these.

Answer 6

I know I am running the risk of sounding like the guy who questions the question instead of answering it, but understand that I am offering this completely in the spirit of trying to be helpful.

In the context of your bigger concern, that is customers are complaining the .net app is slower than VB6, are you seriously considering downconverting if you find out that VB6 is indeed quicker? If not, why waste the time with testing?

Even if you can conclusively prove your customers wrong and .net is faster, you don't really win anything by dismissing customer complaints. Spend the time solving the problem rather than debating the causes, in the end everyone will be happier.

Finally, I will note that switching out platforms to get better performance is generally not a productive endeavor and is usually a last ditch effort by programmers who let platform wars win out over common sense. My suggestion is ignore the .NET/VB6 performance comparison for a project that is already built and just spend the time finding out the specific areas of concern for the application and optimize as best you can.

Answer 7

Oh, it's easy to see. Copy and paste 50 buttons in a form both in WF and in VB6 and compare their paint speed side by side. It is not .NET being slow, it is WF being slow. I never really nailed it down but contributing factors:

• The Button class paints really slow. It asks the container to paint the background for transparency it doesn't have.
• VB6 has several windowless controls, like Label. In WF every control is a window.
• Visual Styles don't come for free.
• They worked really hard on making VB1 work well on a 386SUX.

Answer 8

The .Net fans have a thousand excuses. At one point Microsoft was prohibiting comparisons, probably with reason.

Part of the reason .Net is slow is the runtime startup cost. Part of it is time required to JIT compile. Part of it is garbage collection. Part of it is the overhead of having so much memory consumed, which can lead to more swapping. Part of it is that outside of inline code (i.e. OS calls, many standard libraries) involves thunking through COM and or Win32 layers to get things done.

Newer machines with vast memory and multiple fast CPUS and hard drive speeds masks some of this though. Masking doesn't make it less wasteful.

.Net was targeted at the same things Java is decent at. Fast-written throwaway code and server-side stuff that stays loaded and running for long intervals.