tags:

views:

664

answers:

6

Hi

I have an application with a file receive location. After the host instance has been running for a few hours the receive location fails to identify new files dropped into the folder that it is monitoring. It doesn't forget about them altogether, it's just that performance grinds to a crawl. The receive location is configured to poll the target folder every 60 seconds but after host instance has been running for an hour or so, then it seems that the target folder is being polled only every thirty minutes. If I restart the host instance then the files waiting in the target folder are collected right away and performance is fine for the next hour or so.

The same application runs fine in a different environment. There are now obvious entries in the event log related to the problem. All the BizTalk SQL jobs are running fine except for Backup BizTalk Server (BizTalkMgmtDb).

Any suggestions gratefully received.

Thanks

Rob

+1  A: 

Without more details, the biggest tell is that your Backup Job is failing. If the backup job is failing, it may not be properly configured. If it is properly configured and still failing, then you've got other issues. Can you give us some more information about your BizTalk install.

  1. What version are you running?
  2. What are our database sizes?
  3. What are your purge and archive settings like?
  4. Is there any long running blocks in your SQL Server DB coming from BizTalk?
ChrisLoris
Hi Chris, We're running BTS2006 enterprise on a two node BizTalk group with SQL Server 2005. The group's BTS databases are installed on a different server. All servers are high spec, the BTS boxes are quad CPU with 42gb RAM, the SQL box is 8cpu with 84gb Ram. Both BTS and SQL boxes run almost idle.All BizTalk jobs are now configured and run without a problem. The tracking db is currently 9gb data, 1gb transaction log. The MsgBox db is currently 4gb data, 4gb transaction log.There are no long running blocks
Rob Bowman
+1  A: 

Another thing to consider is the user accounts the send, receive and orchestration hosts are running under. Please check the BizTalk Administration Console. If they are all running the same account, sometimes the orchestrations can starve the send and receive processes of CPU time. I believe priority is given to orchestrations then receive, then send. Even if you are just developing, it is useful to use separate accounts for this. This also improves security.

The Wrox BizTalk Server 2006 will also supply tuning advice.

QSmienk
We have separate host instances for receive, orchestration and send. Although each of these use the same domain accounts I don't think this would cause an issue?
Rob Bowman
+1  A: 

What other things are going on with the server? Is BizTalk pegged otherwise or is it idle?

yieldvs
It is otherwise idle
Rob Bowman
+1  A: 

You mention that the solution does not have any problems in another environment, so it's likely that there is a configuration problem.

Check the following:

** On SQL Server, set some upper memory limit for SQL Server. By default, SQL Server uses whatever it can get and then hangs onto it, so set a reasonable limit so that your system can operate without spending a lot of time paging memory onto and from your hard drive(s).

** Ensure that you have available disk space - maybe you are running low - this can lead to all kinds of strange problems.

** Try to split up the system's paging file among its physical drives (if you have more than one drive on the system). Also consider using a faster drive, or if you have lots of cash laying around, get a SAN.

** In BizTalk, is tracking enabled? If so, are you also tracking message bodies? Disable tacking or message body tracking and see if there is a difference.

** Start performance monitor and monitor the following counters when running your solution

  • Object: BizTalk Messaging
  • Instance: (select the receiving host) %%
  • Counter: Documents Received/Sec

  • Object: BizTalk Messaging

  • Instance: (select the transmitting host) %%
  • Counter: Documents Sent/Sec

  • Object: XLANG/s Orchestrations

  • Instance: (select the processing host) %%
  • Counter: Orchestrations Completed/Sec.

%% You may have only one host, so just use it. Since BizTalk configurations vary, I am using generic names for hosts.

The preceding counters monitor the most basic aspects of your server, but may help to narrow down places to look further. You can, of course, add CPU and Memory too. If you have time (days...maybe weeks) you could monitor for processes that allocate memory and never release it. Use the following counter...

  • Object: Memory
  • Counter: Pool Nonpaged Bytes

Slow decline of this counter indicates that a process is not releasing memory, which affects everything on the system.

Let us know how things turn out!

Erik Westermann
+1  A: 

Here are some additional tools which may help you identify and diagnose BizTalk database issues.

BizTalk MsgBox Viewer

Here is a tool to repair identified errors:

Terminator

Use at your own risk... read the glogs and docs. Start with the message box viewer and let us know our results.

ChrisLoris
I don't think the issue is related to the msgbox. A thorough health check of the BizTalk SQL server shows no problems. Although we have this application running ok in different environment, the environment where we are experiencing the problem is the only one that is subjected to load testing and the only one where the troublesome receive location is configured to received from this particular NAS box
Rob Bowman
Apologies Chris - you were spot on. I ran the MsgBox viewer and could see that the receive host was being throttled because of database size. When I freed up some space for the msgbox and tracking db's the throttling stopped.There's a great blog at http://blogs.msdn.com/biztalkcpr/default.aspx describing process for identifying throttling.Only remaining problem - it seems that I could benefit from using terminator but this can only be run while connnected to the web. Unfortunately I can't connect to the web from any server that has access to the BizTalk db server.
Rob Bowman
Terminator can be run from any machien that has access to the DB. Doesn't need to be run on BizTalk. It's a DB tool in the end. I am having some throttling issues now for other reasons... going to read that blog post. Thanks to you!
ChrisLoris
A: 

Some good suggestions from others. I will add :

Do you have any custom receive pipeline components on the receive location ? If so perhaps one is leaking memory, calling some external component eg database which is taking a long time ?

How big are the files you are receiving ?

On the File transport properties of your receive location, set "file renaming" on, do the files get renamed within 60s.

bamboowave