Article ID: 282791
Article Last Modified on 12/3/2007
APPLIES TO
- Microsoft Windows Server 2003, Standard Edition (32-bit x86)
- Microsoft Windows Server 2003, Enterprise Edition (32-bit x86)
- Microsoft Windows Server 2003, Datacenter Edition (32-bit x86)
- Microsoft Windows Advanced Server, Limited Edition
- Microsoft Windows 2000 Server
- Microsoft Windows 2000 Advanced Server
- Microsoft Windows 2000 Datacenter Server
- Microsoft Windows Small Business Server 2003 Premium Edition
- Microsoft Windows Small Business Server 2003 Standard Edition
This article was previously published under Q282791
SYMPTOMS
When you use disk defragmentation utilities to optimize logical drives that host File Replication service (FRS) replicated content, the following symptoms may occur:
- Files in System Volume (SYSVOL) and Distributed File System (DFS) shares are replicated excessively and there is no apparent change to the files.
- Files may replicate at offpeak hours but at regularly occurring times if you schedule disk defragmentation to run during specific times or periods of low server usage.
- The number of files in the staging folder constantly grows, and then empties after the disk defragmentation utility is completed, or the FRS schedule opens to allow replication.
- The number of files in the staging folder constantly grows but never empties if changes to downstream partners cannot be replicated either because of network connectivity or some other error condition.
- Network traffic between replication partners is consuming excessive network bandwidth as a result of FRS.
CAUSE
In Windows 2000, the file system APIs that support defragmentation use the file cache to do a mapped file write of the file data to the new set of assigned disk clusters. Because a mapped file may not be associated with any particular file handle, the NTFS journal code forces the SourceInfo field in the USN record to zero. This behavior occurs because defragmentation is an on-line operation (the volume is not locked) and the file cache may actually contain modified data for a given byte range of the file if a user was writing to the file while the defragmentation process was underway. However, NTFS cannot tell that this operation has occurred. FSCTL_MARK_HANDLE becomes a no-op. When you stop FRS while Diskkeeper is being run, NTFS still writes the journal records and FRS reads them when it is later restarted.
In Windows Server 2003, NTFS no longer passes the data through the file cache and no USN records are generated.
RESOLUTION
You can use the FRS debug log files and the outbound log of the FRS database to locate the source of FRS replicating files. To determine if disk defragmentation utilities are causing excessive FRS replication traffic in your environment, use either of the methods described in this section.
Method One: Searching the FRS Debug Logs
To search the FRS debug logs to determine the reason for replication:
- Open a Windows NT CMD prompt, and then type: cd /d %systemroot%\debug
- From the %SystemRoot%\Debug folder, use either of the following steps:
- If you are using FRS in Windows 2000 or Windows 2000 Service Pack 1 (SP1), run the following command:
findstr /i "contentcmd" %systemroot%\debug\ntfrs_00??.log
- If you are using FRS in Windows 2000 Service Pack 2 (SP2), Windows Server 2003, increase log level verbosity to 4, and then let the service run for a sufficient period of time to capture replication events. To filter the FRS change events, run the following command:
findstr /i "content" %systemroot%\debug\ntfrs_00??.log
NOTE: Run this command if you are using the hotfix that is described in the following Microsoft Knowledge Base article:
272567 File Replication Service Improvements in Windows 2000 Service Pack 2
For additional information about increasing the logging levels, click the article number below to view the article in the Microsoft Knowledge Base:
221111 Description of FRS Entries in the Registry
- If you are using FRS in Windows 2000 or Windows 2000 Service Pack 1 (SP1), run the following command:
Use the Layout tab in the CMD sessions properties to set the window size width and height to accommodate FINDSTR output. To start, set the width at 110 characters, the height at 45, and the resolution to 1024 x 768.
The "Content|ContentCmd" string in the FINDSTR output displays the change that took place for files on NTFS formatted partitions. The FINDSTR output for files replicated by disk defragmentation utilities have lines at the "[DatOvrWrt ]" or the [StrmOvrWrt] string. For example:
d:\>FINDSTR /i "ContentCmd" ntfrs_00??.log ---------- D:\WINNT\DEBUG\NTFRS_0001.LOG <ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ] <ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [StrmOvrWrt ] <ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ] <ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ] <ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ] <ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ] <ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [StrmOvrWrt ] <ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ] <ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ] <ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ] <ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ] <ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ] <ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd c0/ 0, 00000001 Flags [DatOvrWrt ] <ChgOrdRetryWorker: 648: 11585: S4: HH:MM:SS> ContentCmd
Flags set in the NTFS change log describe modifications to FRS-replicated files. FRS replication is predicated on closed files that reside on NTFS 5.0 formatted partitions in FRS replicated folders. The reasons for changes in FRS are displayed in the following table:
Close - Change log close record
Create - File or folder was created
Delete - File or folder was deleted
RenNew - File or folder was renamed
DatOvrWrt - Main file data stream was overwritten
DatExt - Main file data stream was extended
DatTrunc - Main file data stream was truncated
Info - Basic info change (attrib, last write time, etc.)
Oid - Object Id change
StreamNam - Alternate data stream name change
StrmOvrWrt - Alternate data stream was overwritten
StrmExt - Alternate data stream was extended
StrmTrunc - Alternate data stream was truncated
EAChg - Extended file attribute was changed
Security - File access permissions changes
IndexableChg - File change requires re-indexing.
HLink - Hardlink change
CompressChg - File commpression attribute changed
EncryptChg - File encryption changed
Reparse - Reparse point changed
To increase the log endurance and verbosity for FRS, modify the following registry entries:
- Set the Debug Maximum Log Messages (REG_DWORD) registry entry to 20,000 (do not enter the comma).
- Set the Debug Log Files (REG_DWORD) registry entry to a value between 20 and 50.
- For Windows 2000 SP2 and Windows Server 2003 versions of FRS, set the Debug Log Severity (REG_DWORD) registry entry to 4 to see Content: strings. If you do not to set the log verbosity to a high enough value, you may be able to filter the debug logs against the "UsnReason:" string to locate the the type of modification that took place.
NOTE: Follow the procedure in this step if you used the hotfix described in the following Microsoft Knowledge Base article:272567 File Replication Service Improvements in Windows 2000 Service Pack 2
For additional information about increasing the logging levels, click the article number below to view the article in the Microsoft Knowledge Base:
221111 Description of FRS Entries in the Registry
Method Two: Searching the NTFRS Outbound Log
To search the NTFRS Outbound Log by using the Ntfrsutl (Ntfrsutl.exe) utility and Iologsum.cmd:
- Identify the computer that is originating excessive updates, and then use Ntfrsutl from the Windows 2000 Resource Kit to dump the FRS outbound log.
To do so, from the Windows NT CMD prompt, type:ntfrsutl outlog >outlog.txt
Use the Iologsum.cmd FRS troubleshooting utility to structure the Outlog file that is created in the preceding step. Iologsum supports the following arguments:
c:\>iologsum.cmd /? Usage: C:\iologsum.cmd [-sort=xxx] datafile Process the output of "ntfrsutl inlog" or "ntfrsutl outlog" and summarize the pending change orders. Sort Keyword Sort the output by: -sort=seqnum Sequence Number (default) -sort=version File Version Number -sort=filename File Name -sort=size File Size -sort=fileguid File Guid -sort=origguid Originator Guid -sort=eventtime Event Time
You can use the Iologsum switches to identify the sources of replication. Use the -sort=eventtime switch to identify patterns of replication changes, for example, changes that occur in a scheduled job or process. Use the following command line to sort the Outlog.txt file by event time:
c:\>iologsum -sort=eventtime outlog.txt
The following text is an example of output from an outlog that is sorted by eventtime:
Event Time Ver OrigiGUID File Replication Reason 11/11/2000 3:03 4 646d2289 FILE001.TXT [Content DIR MD5/ StrmOvrWrt ] 11/11/2000 3:03 5 646d2289 FILE002.TXT [Content DIR MD5/ StrmOvrWrt ] 11/11/2000 3:03 4 646d2289 FILE003.TXT [Content DIR MD5/ StrmOvrWrt ] 11/11/2000 3:03 4 646d2289 FILE004.TXT [Content DIR MD5/ StrmOvrWrt ] 11/11/2000 3:03 6 646d2289 FILE005.TXT [Content DIR MD5/ StrmOvrWrt ] 11/11/2000 3:03 5 646d2289 FILE006.TXT [Content DIR MD5/ StrmOvrWrt ] 11/11/2000 3:03 4 646d2289 FILE007.TXT [Content DIR MD5/ StrmOvrWrt ] 11/11/2000 3:03 5 646d2289 FILE008.TXT [Content DIR MD5/ StrmOvrWrt ] 11/11/2000 3:03 4 646d2289 FILE009.TXT [Content DIR MD5/ StrmOvrWrt ] 11/11/2000 3:03 4 646d2289 FILE010.TXT [Content DIR MD5/ StrmOvrWrt ] 11/11/2000 3:03 4 646d2289 FILE011.TXT [Content DIR MD5/ StrmOvrWrt ] 11/11/2000 3:03 4 646d2289 FILE012.TXT [Content DIR MD5/ StrmOvrWrt ] ... 11/11/2000 6:00 2 0c88c766 FILE101.TXT [Content MD5/ DatOvrWrt ] 11/11/2000 6:00 4 0c88c766 FILE102.TXT [Content MD5/ DatOvrWrt ] 11/11/2000 6:00 10 0c88c766 FILE103.TXT [Content MD5/ DatOvrWrt ] 11/11/2000 6:00 2 0c88c766 FILE104.TXT [Content MD5/ DatOvrWrt ] 11/11/2000 6:00 3 0c88c766 FILE105.TXT [Content MD5/ DatOvrWrt ] 11/11/2000 6:00 2 0c88c766 FILE106.TXT [Content MD5/ DatOvrWrt ]
The Iolog output shows a pattern of replication at 3:03AM and 6:00 AM that originates from replica members 646d2289 and 0c88c766 (these numbers can be resolved to computer names in the Ntfrsutil sets output). The reason for the changes are [StrmOvrWrt ] and [DatOvrWrt]. The version columns show that the same files have been replicated anywhere from two to ten times. Files that have high version numbers may illustrate that excessive updates are occurring.
Workarounds to Excessive Replication by Disk Defragmentation Utilities
To prevent disk defragmentation problems from causing excessive replication of FRS files:
- Do not run defragmentation utilities on volumes that contain FRS replicated files. In the short term, disable disk defragmentation routines to stop FRS updates from generating and allow the staging folders to empty.
- Exclude FRS-replicated folders from being optimized by defragmentation utilities. Diskkeeper allows administrators to exclude folders from fragmentation.
- Remove the member computer from all replica sets hosted by the volume that is being defragmented, run the defragmentation utility, and then add the member computer back to the replica sets. When you complete this procedure, you force a non-authoritative restore (D2) but no file data is moved over the network.
FRS can generate the staging file along with a new MD5 checksum. If the checksum matches the value in the IDTable, then abort the local change order. FRS does a lot of work to generate a staging file for each file that gets defragmented, but it does not replicate the file if the file has not changed.
MORE INFORMATION
FRS is a multiple-threaded, multiple-master replication engine that replaces the LanMan Replication (LMREPL) service in Microsoft Windows NT versions 3.x and 4.0. Windows 2000 domain controllers and servers use FRS to replicate system policy and logon scripts for Windows 2000 and down-level clients that reside in the SYSVOL.
FRS can also replicate content between Windows 2000-based servers that host the same fault-tolerant DFS roots or child node replicas. FRS replicates files based on version changes or changes to permissions on files and replicated folders.
Additional query words: `
Keywords: kbproductlink kbnetwork kbprb KB282791
