Knowledge Base

Remote Assistance connection to Windows Server 2003 with FIPS encryption does not work

Article ID: 811770

Article Last Modified on 9/26/2007

APPLIES TO

Microsoft Windows Server 2003, Standard Edition
Microsoft Windows Server 2003, Enterprise Edition
Microsoft Windows Server 2003, Datacenter Edition (32-bit x86)
Microsoft Windows Server 2003, Web Edition
Microsoft Windows Server 2003, 64-Bit Enterprise Edition
Microsoft Windows Server 2003, 64-Bit Datacenter Edition
Microsoft Windows XP Professional

SYMPTOMS

Microsoft has added the FIPS Compliant setting to the options for Terminal Services encryption levels in Windows Server 2003. A Windows Server 2003-based server that has the encryption level set to FIPS Compliant cannot allow Remote Assistance connections from a computer that is running Windows XP, Windows XP Service Pack 1 (SP1), or Windows XP Service Pack 2 (SP2).

When you try to connect from a Windows XP-based client to a Terminal Services server, the connection may not succeed, and you may receive the following error message:

Because of a security error, the client could not connect to the terminal server. After making sure that you are logged on to the network, try connecting to the server again.

CAUSE

Windows XP does not support the FIPS Compliant encryption level. Therefore, a Windows XP-based computer cannot connect to a Windows Server 2003-based server for remote assistance. Additionally, a Windows XP-based computer cannot provide a Remote Assistance connection to a Windows Server 2003-based computer that is configured to require FIPS-compatible encryption.

RESOLUTION

To resolve this problem, install Remote Desktop Connection 6.0. For more information about Remote Desktop Connection, click the following article number to view the article in the Microsoft Knowledge Base:

925876 Remote Desktop Connection (Terminal Services Client 6.0)

WORKAROUND

Remote Desktop Connection (Terminal Services Client 6.0) can be installed on client computers that are running Windows XP SP2.

To work around this problem in Windows XP or in Windows XP SP1, disable the FIPS encryption level. To disable the FIPS encryption level, you can change the Encryption level setting in the RDP-Tcp Properties dialog box, or you can use the Group Policy Object to disable FIPS data encryption system-wide. To disable the FIPS encryption level, use one of the following methods.

Note There are two ways to enable the FIPS encryption level. If you have to disable the FIPS encryption level for Terminal Services, you must do this by using the same method that you originally used to enable the FIPS encryption level.

Method 1

To disable the FIPS encryption level by changing the Encryption level setting in the RDP-Tcp Properties dialog box, follow these steps:

Click Start, click Run, type tscc.msc in the Open box, and then click OK.
Click Connections, and then double-click RDP-Tcp in the right pane.
In the Encryption level box, click to select a level of encryption other than FIPS Compliant.

Note If the Encryption level setting is disabled when you try to change it, the system-wide setting for System cryptography: Use FIPS compliant algorithms for encryption, hashing, and signing has been enabled, and you must disable this system-wide setting by using method 2.

Method 2

To use the Group Policy Object to disable FIPS data encryption system-wide, follow these steps:

Click Start, click Run, type gpedit.msc in the Open box, and then click OK.
Expand Computer Configuration, expand Windows Settings, expand Security Settings, expand Local Policies, and then click Security Options.
In the right pane, double-click System cryptography: Use FIPS compliant algorithms for encryption, hashing, and signing, click Disable, and then click OK.

Note Encryption level settings in Terminal Server are unavailable when FIPS is enabled.

For more information about scoping Group Policy Objects and troubleshooting the resultant policies on your server, click the following article number to view the article in the Microsoft Knowledge Base:

818735 White Paper: Administering Group Policy by Using the Group Policy Management Console

For more information about the GPO setting for System cryptography: Use FIPS compliant algorithms for encryption, hashing, and signing, click the following article number to view the article in the Microsoft Knowledge Base:

811833 The effects of enabling the "System cryptography: Use FIPS compliant algorithms for encryption, hashing, and signing" security setting in Windows XP and later versions

STATUS

Microsoft has confirmed that this is a problem in the Microsoft products that are listed in the "Applies to" section.

MORE INFORMATION

The FIPS Compliant setting requires that all data between the client and the server be encrypted by using encryption methods that are validated by Federal Information Processing Standard 140-1. When a Windows XP-based client tries to connect to a Windows Server 2003-based server that requires FIPS-compliant encryption, the following errors occur:

On the client, you receive the following error message from Remote Assistance:

A Remote Assistance connection could not be established. You may want to check for network issues or determine if the invitation expired or was cancelled by the person who sent it.
The following error is logged in the System log on the server:

Event ID: 50
Source: TermDD
Type: Error
Description: The RDP protocol component "DATA ENCRYPTION" detected an error in the protocol stream and has disconnected the client.

Additional query words: protection

Keywords: kbqfe kbprb kbhotfixserver KB811770

Microsoft KB Archive/811770

Contents