Microsoft KB Archive/254949

From BetaArchive Wiki
Knowledge Base

IPSec support for client-to-domain controller traffic and domain controller-to-domain controller traffic

Article ID: 254949

Article Last Modified on 10/12/2007


  • Microsoft Windows Server 2003, Standard Edition (32-bit x86)
  • Microsoft Windows 2000 Server
  • Microsoft Windows 2000 Advanced Server
  • Microsoft Windows 2000 Professional Edition
  • Microsoft Windows XP Professional

This article was previously published under Q254949

Important This article contains information about how to modify the registry. Make sure to back up the registry before you modify it. Make sure that you know how to restore the registry if a problem occurs. For more information about how to back up, restore, and modify the registry, click the following article number to view the article in the Microsoft Knowledge Base:

256986 Description of the Microsoft Windows registry


This article describes the supported configurations for using Internet Protocol security (IPSec) to encrypt network traffic from a client computer to a domain controller or from a domain controller to another domain controller.


Important The information in this section applies only to those products listed in the "Applies to" section.

We support the use of IPSec to encrypt network traffic in end-to-end client-to-client, client-to-server, and server-to-server implementations when you use either Kerberos computer authentication or when you use certificate-based computer authentication. Currently, we do not support the use of IPSec to encrypt network traffic from a domain client or member server to a domain controller when you apply the IPSec policies by using Group Policy or when you use the Kerberos version 5 protocol authentication method.

Additionally, we support using IPSec to encrypt both the following kinds of network traffic:

  • Domain controller-to-domain controller replication traffic
  • Global catalog-to-global catalog replication traffic

To encrypt this traffic by using IPSec, configure both the following:

  • Create an IPSec policy filter to encrypt all Unicast traffic by using the All IP Traffic option.
  • Configure this IPSec policy filter to encrypt this traffic between two IP addresses by using IPSec Transport mode. In this scenario, do not use IPSec Tunnel mode.

Warning Serious problems might occur if you modify the registry incorrectly by using Registry Editor or by using another method. These problems might require that you reinstall your operating system. Microsoft cannot guarantee that these problems can be solved. Modify the registry at your own risk.

After you configure this IPSec policy, you may notice that when the computers are started, several packets may be sent over the network unencrypted. This issue occurs because some packets might be sent over the network before the IPSec driver has been initialized and before the IPSec policy has been processed. To resolve this issue, put the IPSec driver IPSec.sys into Block Mode during the computer startup process. When you do this, IPSec blocks outgoing network traffic from the computer until the PolicyAgent component starts and until the PolicyAgent component loads the IPSec policies. After the IPSec PolicyAgent component has started, and after the IPSec policies are loaded, the PolicyAgent changes the IPSec driver's operation mode to permit the passage of IPSec traffic. To put the IPSec driver into Block Mode, set the following registry value:


Value name: OperationMode
Value type: REG_DWORD
Value data: 1

A value of 1 puts the IPSec driver into Block Mode. A value of 0 (zero) bypasses the IPSec driver's block mode.

For more information, click the following article number to view the article in the Microsoft Knowledge Base:

254728 IPSec does not secure Kerberos traffic between domain controllers

We support using IPSec to encrypt domain controller-to-domain controller traffic such as Server Message Block (SMB), Remote Procedure Call (RPC) replication, and other kinds of traffic. You can transport this traffic by using IPSec to let you easily pass these kinds of traffic through a firewall. In this scenario, you only have to permit IPSec traffic and Internet Key Exchange (IKE) traffic through your firewall. For more information, click the following article number to view the article in the Microsoft Knowledge Base:

233256 How to enable IPSec traffic through a firewall

We recommend that you require certificate-based authentication when you configure domain controller-to-domain controller IPSec policy rules. For detailed information about how to create an IPSec policy, see the Active Directory in Networks Segmented by Firewalls document. To obtain this document, visit the following Microsoft Web site:

The rule must require certificate authentication if the security requirements do not allow Kerberos traffic through the firewall. By default, IKE certificate revocation checking is off, and may have to be enabled through the firewall. This depends on the PKI infrastructure that is being used.

Build the IPSec rule on the domain controllers by using the following specifications:

  • The filter list specifies traffic going from the IP address on DC1 to the IP address on DC2 (mirrored), subnet masks, all protocols, and all ports. You may want to add a rule to the IPSec policy to exempt ICMP traffic from IPSec security negotiation if Ping is used to verify network connectivity to the remote system through the firewall. Otherwise, connectivity can be verified by a network sniff that shows IKE traffic (ISAKMP, UDP port 500) being sent and received from the DC to the other DC IP address.

A network address translator (NAT) must not be used to change addresses or modify packets between the domain controllers that require IPSec protection between them.

  • Under Tunnel Setting, click This rule does not specify an IPSec tunnel so that it uses Transport mode.
  • Select Use Certificate for the authentication method. You can use Kerberos, see the following note.
  • Create a custom filter action by clearing the Accept Unsecured Communication and Allow Unsecured Communication check boxes and specifying the appropriate data encryption method by using the ESP format of IPSec. Network adaptors that perform IPSec per-packet encryption in hardware are needed in each domain controller so that IPSec encryption does not consume all the computer's CPU cycles.

Note The initial release (build 2195) of Windows 2000 does not protect IKE, Kerberos, or RSVP traffic using IPSec transport filters. If Kerberos is used as the IPSec rule authentication method to protect domain controller-to-domain controller traffic instead of certificates, the firewall also must allow Kerberos traffic to go through. This must be a default setting. Windows 2000 Service Pack 1 provides IPSec with the capability of protecting Kerberos and RSVP traffic.

253169 Traffic that can--and cannot--be secured by IPSec

Keywords: kbinfo kbipsec kbnetwork KB254949