Article ID: 918643
Article Last Modified on 10/15/2007
APPLIES TO
- Microsoft BizTalk Server 2006 Enterprise Edition
- Microsoft BizTalk Server 2006 Developer Edition
- Microsoft BizTalk Server 2006 Standard Edition
- Microsoft BizTalk Server 2004 Enterprise Edition
- Microsoft BizTalk Server 2004 Developer Edition
- Microsoft BizTalk Server 2004 Partner Edition
- Microsoft BizTalk Server 2004 Standard Edition
- Microsoft BizTalk Server 2002 Standard Edition
- Microsoft BizTalk Server 2000 Standard Edition
Important This article contains information about how to modify the registry. Make sure that you back up the registry before you modify it. Make sure that you know how to restore the registry if a problem occurs. For more information about how to back up, restore, and modify the registry, click the following article number to view the article in the Microsoft Knowledge Base:
322756 How to back up and restore the registry in Windows XP and Windows Vista
SUMMARY
Memory leaks are a general issue. You may have to try several steps to find the specific cause of a memory leak or an out-of-memory (OOM) exception in Microsoft BizTalk Server. This article discusses important things to consider when you are evaluating memory usage and possible memory-related issues. These considerations include the following:
Physical RAM
Large message processing
Use of the /3GB switch
Use of custom components
Which version of the Microsoft .NET Framework the system is running
The number of processors the system has
This article also discusses known issues that affect memory usage in BizTalk Server 2004 and in BizTalk Server 2002 Service Pack 1 (SP1).
Additionally, this article discusses common causes and resolutions for memory leaks and out-of-memory exceptions in Microsoft BizTalk Server 2006 R2, in BizTalk Server 2006, and in BizTalk Server 2004, together with tools that you may use to identify the cause of your issue.
INTRODUCTION
This article describes how to troubleshoot a memory leak or an out-of-memory exception in the BizTalk Server process of Microsoft BizTalk Server.
MORE INFORMATION
The BizTalk Server process may be experiencing a memory leak when memory usage in Microsoft Windows Task Manager consumes more than 50 percent of the physical RAM. A memory leak may cause an out-of-memory exception when memory usage increases until the process runs out of system memory or until the process stops functioning.
Note In Microsoft BizTalk Server 2006 R2, in BizTalk Server 2006, and in BizTalk Server 2004, the BizTalk Server process is the Btsntsvc.exe process. In BizTalk Server 2002 and in BizTalk Server 2000, the BizTalk Server process is the Mscis.exe process.
When this problem occurs, a warning message that resembles the following message is logged in the event log:
Event Type: Warning
Event Source: BizTalk Server 2004
Event Category: (1)
Event ID: 5410
Date: 3/6/2006
Time: 5:00:11 PM
User: N/A
Computer: <ComputerName>
Description: An error has occurred that requires the BizTalk service to terminate. The most common causes are an unexpected out of memory error and an inability to connect or a loss of connectivity to one of the BizTalk databases. The service will shutdown and auto-restart in 1 minute. If the problematic database remains unavailable, this cycle will repeat.
Error message: Exception of type System.OutOfMemoryException was thrown.
Error source:
BizTalk host name: BizTalkServerApplication
Windows service name: BTSSvc{DCC899FE-C62F-41BE-851A-8720B2EB9C14}
Event Type: Warning
Event Source: BizTalk Server 2006
Event Category: (1)
Event ID: 5410
Date: 9/11/2007
Time: 10:11:39 AM
User: N/A
Computer: <ComputerName>
Description: An error occurred that requires the BizTalk service to terminate. The most common causes are the following: 1) An unexpected out of memory error. OR 2) An inability to connect or a loss of connectivity to one of the BizTalk databases. The service will shutdown and auto-restart in 1 minute. If the problematic database remains unavailable, this cycle will repeat.
Error message: Exception of type 'System.OutOfMemoryException' was thrown.
Error source: mscorlib
BizTalk host name: BizTalkServerApplication
Windows service name: BTSSvc$BizTalkServerApplication
Note In these messages, ComputerName represents the actual computer name.
Important considerations
Physical RAM and memory usage
Because it may be expected behavior for a process to use about half the physical RAM, use the memory usage as a guideline. For example, if the computer that is running BizTalk Server has 2 gigabytes (GB) of RAM, and the BizTalk Server process uses about 500 megabytes (MB) of RAM, there may not be leak. The memory consumption may be caused by a long-running stored procedure or orchestration. Make sure that you know what behavior is expected in your environment to determine whether a memory leak is occurring.
Large messages
When BizTalk Server processes large messages, the system may appear to have a memory leak. However, the messages are just using lots of memory. For more information, visit the following Microsoft Developer Network (MSDN) Web site:
Also, consider that high memory usage may be expected if BizTalk Server is processing large messages. You may want to upgrade your hardware to meet the performance requirements of BizTalk Server in your environment.
How long it takes to reproduce the memory leak
Memory leaks can occur immediately or they may accumulate over time. Both scenarios are common.
Use of the /3GB switch on 32-bit computers
When the /3GB switch is enabled, a process can access 3 GB of virtual address space that is reserved for user mode operation. (Typically, a process can access 2 GB of virtual address space.) The /3GB switch is an option for systems that require more addressable memory. This option may improve memory usage for processing messages. However, the /3GB switch allows for only 1 GB of addressable memory for kernel mode operations. Additionally, this switch may increase the risk of running out of pool memory.
For more information about the /3GB switch, visit the following Microsoft Developer Network (MSDN) Web site:
The following table lists PAE and 3GB supportability for BizTalk Server 2002, for BizTalk Server 2004, for BizTalk Server 2006, and for BizTalk Server 2006 R2.
| Product | PAE | 3GB |
|---|---|---|
| BizTalk 2002 | No | No |
| BizTalk 2004 | Yes | No |
| BizTalk 2006 | Yes | Yes |
| BizTalk 2006 R2 | Yes | Yes |
If you must enable the /3GB switch to meet the performance requirements of a computer that is running BizTalk Server 2006 R2 or BizTalk Server 2006, you may want to consider adding servers to your environment. If you are concerned about user mode memory on a computer that is running BizTalk Server 2002, you may want to consider using multiple hosts.
BizTalk components that run inside a Microsoft Internet Information Services (IIS) process may also benefit when the /3GB switch is enabled. These components include IIS 6.0 in Windows Server 2003 , IIS 5.1 in Windows XP, IIS 5.0 Isolation Mode in Windows Server 2003, and IIS 5.0 in Microsoft Windows 2000.
The /3GB switch is not supported on computers that are running Windows SharePoint Services 2.0 or later versions or SharePoint Portal Server 2003 SP2 or later versions. For more information, click the following article number to view the article in the Microsoft Knowledge Base:
933560 The Windows Server 2003 /3GB switch is not supported in Windows SharePoint Services 2.0 or in later versions or in SharePoint Portal Server 2003 Service Pack 2 or in later versions
The /3GB switch also increases the maximum private bytes of the BizTalk process from 800 MB to 1800 MB. For more information about .NET application performance with the /3GB switch enabled, visit the following Microsoft Developer Network (MSDN) Web site:
Note BizTalk Server 2002, BizTalk Server 2004, BizTalk Server 2006, and BizTalk Server 2006 R2 benefit from the 4 GB memory address space when you use a 64-bit computer.
Use of custom components
If you are using custom components, such as pipelines, Application Integration Components (AIC) for BizTalk Server 2002, or service components, together with BizTalk Server, you must know what these components are doing. You must also know the potential effect of these components on memory usage. A common memory problem occurs when a component is transforming a document. The transform operation is a memory-intensive operation. When a document is transformed, BizTalk Server passes the message stream to the Microsoft .NET Framework XslTransform class in BizTalk Server 2006 R2, in BizTalk Server 2006, and in BizTalk Server 2004. BizTalk Server passes the message stream to the XML Document Object Model (DOM) in BizTalk Server 2002 and in BizTalk Server 2000.
Another common issue occurs when there is intensive string manipulation. Intensive string manipulation can consume lots of memory. For more information about ways to improve performance, visit the following MSDN Web site:
Version of the .NET Framework
The Microsoft .NET Framework 2.0 and the .NET Framework 1.1 have different memory behavior. Therefore, you may see varying results between them. If you are using the .NET Framework 1.1, confirm that the .NET Framework 1.1 Service Pack 1 (SP1) is installed. SP1 addresses many memory issues. For more information, click the following article number to view the article in the Microsoft Knowledge Base:
867460 List of bugs that are fixed in the .NET Framework 1.1 Service Pack 1
Number of processors
The common language runtime (CLR) has the following garbage collectors (GCs):
- Workstation (Mscorwks.dll)
- Server (Mscorsvr.dll)
If the computer that is running BizTalk Server is a multiprocessor system, the .NET Framework uses the Server version of the execution engine. This is the default behavior. The Server garbage collector is designed for maximum throughput. Additionally, the Server garbage collector scales to provide very high performance. This garbage collector allocates memory and then later frees memory to provide high performance on the system. Therefore, a computer that is running BizTalk Server together with some .NET Framework components may appear to have a memory leak. However, in this scenario, high memory usage is the expected behavior. If the computer runs out of system memory, or if the process stops working because of insufficient addressable memory, a memory leak condition may exist.
If the computer that is running BizTalk Server is a single processor system, the .NET Framework uses the Workstation version of the execution engine. This is the default behavior. The Workstation garbage collector allocation algorithm is not designed for scaling or for maximum throughput. This garbage collector uses concurrent garbage collector methods. These methods are designed for applications that have complex user interfaces. Such applications may require more aggressive garbage collection.
Warning Serious problems might occur if you modify the registry incorrectly by using Registry Editor or by using another method. These problems might require that you reinstall the operating system. Microsoft cannot guarantee that these problems can be solved. Modify the registry at your own risk.
Sometimes, it may be appropriate to run the Workstation version of the execution engine on a multiprocessor system. You can use the following registry key to switch to the Workstation version of the execution engine:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\BTSSvc{7F09E353-4EFC-440B-249D-9D4ED58795FB}\
Value name: Flavor
Value type: String
Value data: wks
For more information, visit the following MSDN Web sites:
http://msdn2.microsoft.com/en-us/library/ms973838
http://msdn.microsoft.com/msdnmag/issues/1200/GCI2/default.aspx
Known issues
BizTalk Server 2004
If you are using BizTalk Server 2004, you may experience some memory-related issues. To resolve these issues, apply the hotfixes that are described in the following Microsoft Knowledge Base (KB) articles:
899425 FIX: Out of memory exception errors may occur in the BizTalk Server process (Btsntsvc.exe) in Windows Server 2003
910142 FIX: The BizTalk Service may appear to stop responding and you may receive out-of-memory error messages in BizTalk Server 2004
BizTalk Server 2002 Service Pack 1
If you use Microsoft BizTalk Server 2002 Service Pack 1 (SP1), you may experience some memory-related issues. To resolve these issues, apply the hotfix that is described in the following KB article:
885223 FIX: You may receive a "[0x8007000e] Not enough storage is available to complete this operation" error message on your BizTalk Server 2002 computer
Warning Serious problems might occur if you modify the registry incorrectly by using Registry Editor or by using another method. These problems might require that you reinstall the operating system. Microsoft cannot guarantee that these problems can be solved. Modify the registry at your own risk.
If the KB885223 hotfix does not resolve the issue, try changing the value of the HeapDeCommitFreeBlockThreshold registry key. By default, the value of this key is 0. A value of 0 means that the heap manager decommits each 4-kilobyte (KB) page that becomes available. Decommit operations can cause virtual memory fragmentation. The size of the HeapDeCommitFreeBlockThreshold setting in the heap manager will depend on the kind of work that the system is doing. A size of 0x00040000 is a recommended starting value.
Consider the following information before you change the value of the HeapDeCommitFreeBlockThreshold registry key:
- This change only applies to memory fragmentation problems.
- This change is system-wide. Therefore, most processes will use more memory on startup.
- Only consider this change for systems that have BizTalk Server as their primary mission.
To help reduce virtual memory fragmentation, you can increase the size of the HeapDeCommitFreeBlockThreshold setting in the heap manager by changing the value of the following registry key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager
Value name: HeapDeCommitFreeBlockThreshold
Value type: REG_DWORD
Value data: 0x00040000 (This is the recommended starting value.)
Value default: not present
For more information about the HeapDeCommitFreeBlockThreshold registry key, click the following article number to view the article in the KB:
315407 The "HeapDecommitFreeBlockThreshold" registry key
Common causes and resolutions in BizTalk Server 2006 and in BizTalk Server 2004
Transform operations
When BizTalk Server 2006 and BizTalk Server 2004 perform XML transform operations on relatively large messages in a receive port, in a send port, or in XLANG, XSL transforms load the whole message in memory.
To resolve this issue, use one of the following methods:
- Decrease the number of messages that BizTalk Server processes at the same time.
- Decrease the size of the XML message that is being transformed.
The System.Policy.Security.Evidence object is often used in transforms and can consume a lot of memory. Whenever a map contains a scripting functoid that uses inline C# (or any other inline language), the assembly is created in memory. The System.Policy.Security.Evidence object uses the object of the actual calling assembly. This situation creates a rooted object that is not deleted until the BizTalk service is restarted.
Most of the default BizTalk functoids are implemented as inline script. These items can cause System.Byte[] objects to collect in memory. To minimize memory consumption, we recommend that you put any map that uses these functoids into a small assembly. Then, reference that assembly. Use the chart below to determine which functoids use inline script and which functoids do not use inline script.
In the second column, “Yes” means that this functoid is implemented as inline script, and that it will cause System.Byte[] objects to collect in memory. “No” means that this functoid is not implemented as inline script, and that it will not cause System.Byte[] objects to collect in memory.
| Functoids | Inline script? |
|---|---|
| All String Functoids | Yes |
| All Mathematical Functoids | Yes |
| All Logical Functoids except IsNil | Yes |
| Logical IsNil Functoid | No |
| All Date/Time Functoids | Yes |
| All Conversion Functoids | Yes |
| All Scientific Functoids | Yes |
| All Cumulative Functoids | Yes |
| All Database Functoids | No |
| Advanced Functoids | Inline script? |
| Looping Functoid | No |
| Value Mapping Flattening Functoid | No |
| Assert Functoid | No |
| Table Extractor Functoid | No |
| Table Looping Functoid | No |
| Scripting Functoid with Inline C# | Yes |
| Scripting Functoid with Inline JScript.NET | Yes |
| Scripting Functoid with Inline VB.NET | Yes |
| Scripting Functoid with Inline XSLT | No |
| Scripting Functoid with Inline XSLT Call Template | No |
| Scripting Functoid calling External Assembly | No |
| Nil Value Functoid | No |
| Value Mapping Functoid | No |
| Mass Copy Functoid | No |
| Iteration Functoid | No |
| Index Functoid | No |
| Record Count Functoid | No |
BizTalk Server 2006 significantly improves memory management for large documents. To do this, BizTalk Server implements a configurable message size threshold for loading documents into memory during transform operations. The default message size threshold is 1 MB. For more information, visit the following MSDN Web site:
Large attribute values and large element values
When BizTalk Server 2006 and BizTalk Server 2004 execute a receive pipeline or a send pipeline on an XML document, the payload is processed in memory if the document contains one or more of the following entities:
- Large attribute values
- Large element values
- Large attribute or element tags
To resolve this issue, limit the size of these entities. If this method is not possible, make sure that your process does not process multiple documents such as these at the same time.
Custom pipeline components
You are using a custom pipeline component that loads the whole stream into memory. All the components that are included with BizTalk Server, except transforms, support streaming. These components do not use as much memory during streaming. However, custom pipeline components may not support streaming.
Streaming under heavy stress
Send hosts run out of memory when they operate under heavy stress. BizTalk Server send pipelines and send adapters support streaming. In streaming, each component loads a small fragment of the stream into memory. Because each message includes other data structures, together with a message context that can be big or small, this behavior affects the behavior of BizTalk Server under heavy stress.
The behavior of BizTalk Server is affected because the engine loads a pre-configured number of messages. The number of messages that the engine loads is based on the values that appear in the LowWaterMark field and the HighWaterMark field of the Adm_serviceClass table. The Adm_serviceClass table is in the BizTalk Management Database. These values control the number of messages that BizTalk Server processes or sends at the same time.
The HighWaterMark value is the total number of messages that the engine processes at the same time. The default value is 200 messages per CPU. Therefore, on an 8-processor server, the send host will try to process 1,600 messages (200*8) at the same time. If you assume that each message is 50 KB, the messages equal 80 MB (1,600*50=80,000KB).
To resolve this issue, you can change the HighWaterMark value and the LowWaterMark value in the database. The values that you use depend on the size of the messages.
For more information about common causes of an out-of-memory condition, see the "Memory Growth in BizTalk Messaging" section at the following Microsoft Web site:
For BizTalk Server 2006, you can change the default host throttling settings. For more information about how to change the default host throttling settings, visit the following MSDN Web site:
Try to simplify the issue
If you have identified a memory leak, try to determine the cause by removing custom components or by simplifying a map. Also, try to reproduce the issue by using a simple orchestration or a simple solution. Typically, you should create separate receive hosts for receive adapters. You should also create separate send hosts for send adapters. When you use this method, each adapter can run in a separate process. Therefore, if your BizTalk Server host or your BizTalk Server process experiences an out-of-memory condition, you will know which components are involved.
Troubleshooting steps
To troubleshoot an out-of-memory condition, use the Debug Diagnostics tool to monitor memory allocations over time. The Debug Diagnostics tool can create and analyze a memory leak dump file (.dmp). When you troubleshoot memory leaks, the goal is to attach Leaktrack.dll before the high memory condition reproduces to capture memory growth over time. Leaktrack.dll is included with the Debug Diagnostics tool.
- Install the Debug Diagnostics tool. To obtain the Debug Diagnostics tool, visit the following Microsoft Web site:
The Debug Diagnostics tool is part of the Microsoft Internet Information Services (IIS) Diagnostics Toolkit. To install the Debug Diagnostics tool, perform a custom installation. Select only the Debug Diagnostics tool. By default, the installation path of the Debug Diagnostics tool is as follows:
However, you can change the installation path during the installation process.C:\Program Files\IIS Resources\DebugDiag
- Use Performance Monitor to collect data about system performance. This data may provide important indicators about the efficiency of your BizTalk Server environment. The goal is to capture process performance over time. Therefore, enable Performance Monitor logging before the memory leak occurs.
How to use Performance Monitor logging
Select the data to log
To select the data to log, use the method that is appropriate for your operating system:
- For Windows Server 2003 or for Windows XP
- Expand Performance Logs and Alerts.
- Right-click Counter Logs, and then click New Log Settings. The New Log Settings dialog box appears.
- In the Name box, type a descriptive name, and then click OK.
- Note the log file location. (You can also click the Log Files tab, and then click Configure to change the log file location.)
- Click Add Counters.
- Select All counters and All instances.
- In the Performance object list, select the following objects. Click Add after you select each object.
- .Net CLR Exceptions
- .Net CLR Memory
- BizTalk:Messaging
- BizTalk:TDDS
- Memory
- Process
- Processor
- XLANG/s Orchestrations
If SQL Server is local, also add the following objects:
- SQLServer:Databases
- SQLServer:General Statistics
- SQLServer:Memory Manager
- Click Close.
- Change the value in Data Sampling Interval to 5 seconds.
Note The Data Sampling Interval value and the time to start monitoring are subjective. These values depend on when the memory leak is reproduced. Because the log file can be large, specify an interval in which you can obtain the information that you require without overwhelming the server. - Click OK.
- For Windows 2000 Server
- Expand Performance Logs and Alerts.
- Right-click Counter Logs, and then click New Log Settings. The New Log Settings dialog box appears.
- In the Name box, type a descriptive name, and then click OK.
- Note the log file location. (You can also click the Log Files tab to change the log file location.)
- Click Add.
- Select All counters and All instances.
- In the Performance object list, select the following objects. Click Add after you select each object.
- .Net CLR Exceptions
- .Net CLR Memory
- BizTalk:Messaging
- BizTalk:TDDS
- Memory
- Process
- Processor
- XLANG/s Orchestrations
If SQL Server is local, also add the following objects:
- SQLServer:Databases
- SQLServer:General Statistics
- SQLServer:Memory Manager
- Click Close.
- Change the value in Data Sampling Interval to 5 seconds.
Note The Data Sampling Interval value and the time to start monitoring are subjective. These values depend on when the memory leak is reproduced. Because the log file can be large, specify an interval in which you can obtain the information that you require without overwhelming the server. - Click OK.
Obtain the dump file
To obtain the dump file, use one of the following methods:
- Method 1: Automatic
If you know when the memory leak will occur, you can create a Memory and Handle Leak Rule. The Memory and Handle Leak rule automatically attaches Leaktrack.dll.- Click Start, point to Programs, point to IIS Diagnostics (32 bit), point to Debug Diagnostics Tool, and then click Debug Diagnostics Tool 1.0.
- Select Memory and Handle Leak, and then click Next.
- Select Btsntsvc.exe or Mscis.exe, and then click Next.
- On the Configure Tracking Duration page, you can specify a warm-up time. However, if the issue reproduces immediately, select the Start memory tracking immediately when rule is activated check box. For Tracking time, specify the time that will be required to reproduce the issue. At the end of the specified time, a dump file will be created. Make sure that the Auto-create a crash rule to get userdump on unexpected process exit check box is selected.
- Click Next.
- On the Select Dump Location And Rule Name page, click Finish. (You can also change the path of the dump file in the Userdump Location box on this page.)
Note The status is now active. Every time that a dump file is created, the value will increase in the Userdump Count column on the Rules tab in the Debug Diagnostics tool. The default dump file location is C:\Program Files\IIS Resources\DebugDiag\Logs.
- Method 2: Manual
If you do not know when the memory leak will occur, you can manually attach Leaktrack.dll and manually obtain the dump file manually. To do this, follow these steps:- Click Start, point to Programs, point to IIS Diagnostics (32 bit), point to Debug Diagnostics Tool, and then click Debug Diagnostics Tool 1.0.
- Click the Processes tab.
- On the Processes tab, right-click Btsntsvc.exe or Mscis.exe, and then click Monitor For Leaks. The Debug Diagnostics Tool dialog box appears.
- In the Debug Diagnostics Tool dialog box, click Yes, and then click OK.
- When the process reaches 60 percent to 80 percent of RAM, right-click Btsntsvc.exe or Mscis.exe, and then click Create Full Userdump.
Create a crash rule to monitor the same Btsntsvc.exe or Mscis.exe process in case the process terminates before the memory dump is generated:
- Click Start, point to Programs, point to IIS Diagnostics (32 bit), point to Debug Diagnostics Tool, and then click Debug Diagnostics Tool 1.0.
- Select Crash, and then click Next.
- Select A specific process, and then click Next.
- Select the same Btsntsvc.exe or Mscis.exe process, and then click Next.
- On the Advanced Configuration (Optional) page, click Next.
- In the Select Dump Location And Rule Name (Optional) dialog box, click Next.
- Select Activate the rule now, and then click Finish.
Stop Performance Monitor logging
Stop Performance Monitor logging about two minutes after the dump file of the process has been created. To do this, follow these steps:
- Expand Performance Logs and Alerts.
- For each counter log that you created in the "Select the data to log" section, right-click the name of the counter log, and then click Stop.
Analyze the dump file
To determine the cause of a memory leak, you can use the Debug Diagnostics tool to analyze the dump file. To do this, follow these steps:
- Click the Advanced Analysis tab.
- Click Add Data Files, and then locate the .dmp file.
- Select the Memory Pressure Analysis script, and then click Start Analysis.
When the analysis is finished, an analysis report file (.mht) will be created in the C:\Program Files\IIS Resources\DebugDiag\Reports folder. The report file will also be displayed in your browser. The report file contains the results of the analysis. Additionally, the report file contains recommendations for how to resolve the memory leak.
If you use custom DLLs, you can add the symbol path of the custom .pdb files for analysis. To do this, follow these steps:
- Open the Debug Diagnostics tool.
- On the Tools menu, click Options and Settings.
- In the Symbol Search Path For Debugging box, type the symbol path.
If you want help analyzing the dump file, contact Microsoft Product Support Services. For a complete list of Product Support Services telephone numbers and information about support costs, visit the following Microsoft Web site:
Before you contact Product Support Services, compress the dump file, the Performance Monitor log, the analysis report file, and the updated event logs (.evt files). You may have to send these files to a BizTalk Server support engineer.
Keywords: kbhowto KB918643
