Defining SLAs for WCF Services

Question

I have to performance/load test a bunch of interdependant services. They all use net.tcp and most use duplex contracts and internal queueing. [handrolled POCO queue class using lock(syncRoot) { if(queue.Empty) Thread.Wait(); }]

Here's the approach I've come up with:

1. Identify WCF Services to be performance tested
2. Identify the relavant Performance counters for each of the Services
3. Identify the logical startpoint that would take the execution through the services being tested
4. Auto Generate Unit Tests using VS.Net for each of the services
5. Write specific Functional Tests (For example, I can take a use case - "Place an Order" - and write tests that make all the calls to the relevant services and generally excercise pretty much all the functionality needed)
6. Use the trace files from running #5 to generate Unit Tests [using WCF Load Test from CodePlex] (This somehow seems to me an ideal tool for recreating user errors in production/field in a debug environment. Disclaimer: Not used the tool. Impressions from reading the project desc)
7. The tests above could be tweaked to make the calls with auto generated input data
8. Introduce variations to input so different code paths are excercised
9. Log data from performance counters
10. Analyze and identify bottlenecks

Questions:

1. Is there a better approach?
2. In case of services that use internal queues, measuring performance using std performance counters, an issue. I may need custom counters?
3. If #1 is true, is there a way to introduce customer counters without changing code of the services being tested?
4. Should I care about the results of my functional tests?
5. Is there a way to [non-intrusively] implement SLAs for WCF services? (I think if I have enough data from my counters such as requests served, exceptions occurred , response time etc., I should be able to validate my SLA - serve 200,000 requests within 5 mins with a response time of 2 seconds for each request - against these figures. My question perhaps is whether I can just specify my SLA and a product / tool could do all of the plumbing behind the scene and get me a tabulated answer? I know... I know... I was day dreaming :))
6. Aside: What's the best method to queue requests internally in a WCF Service?

Answer 1

Wow...the title was definitely the tip of the iceberg! Hope I'm not way off base here on my responses! :)

1. performance testing of WCF services can be done many different ways: using test tools such as Microsoft Team Test, Borland Silk Performer, Mercury LoadRunner, or something like LoadGen or a custom test harness. My preference is to try to take the approach that you've done in creating some sort of functional unit test and then feeding data into that test, while using the test tool to spin up multiple concurrent instances of that test (virtual 'users'). The tooling of most commercial test tools really facilitate this type of testing so its difficult to go wrong here. The biggest challenge usually grows out of maintaining the test cases and test data to support testing the application.

2. WCF doesn't have any built in counters related to performance. Thats actually a blind spot. Sure, you can see how many connections are being made to the server, but this is coarse information and you probably want to know which services are servicing those requests. Rumor is that Microsoft's 'Dublin', as part of providing a rich hosting environment for WCF/WF, will include surfacing performance counters for the hosted service. We will have to wait and see what actually shakes out.

3. If I needed to instrument a WCF service without impacting the existing codebase I'd look into how much mileage I could get out of a WCF behavior that i could apply to the service. This custom behavior could likely surface the performance counters you might want.

4. Yes. I would care about results (meaning performance?) of a functional test. The caveat is that there is likely some startup (JIT) which can be ignored. I'd probably look to profile the execution of a functional test to get execution metrics - but i'd not turn on code profiling during a performance run.

5. For SLA's again custom behaviors might be the answer. You could log operational metrics to a database and then report off that. Commercial products like Amberpoint and SOA Software will provide support for this as well (including performance counters).

6. Queuing requests to a wcf service? I immediately think of net.msmq binding, especially if your looking to make the request durable.