Technology news and commentary by the staff

Slow System Performance When Copying Large Files in XP x64 & Server 2003 x64 – And a solution!

with 6 comments

I’ve been using Windows XP x64 for about a year now. My previous OS of choice was Windows 2000 which I really loved, I swore to never upgrade to XP and in a way I succeeded because XP x64 is really just Windows Server 2003 x64 with another name. Some apps even identify it as the Server OS since it bears the same NT build number of NT v5.2.

The reason I finally upgraded was so I could access all of my new system’s 8GB of RAM, and also because I was getting tired of writing Win2K fixes for games like Bioshock and Crysis. ;-)
But from the very beginning of the transition I was disappointed; XP x64, when handling large files would slow down to a crawl.

I tried in vain to look for a solution, to give you an example: I’ve got a RAID5 array with 3 x 1TB hard drives and a lone Raptor 150GB for the OS. If I played back a 1080p trailer off Apple’s site stored to my hard drive using Quicktime Alternative the video would start to skip massively if at the same time I copied a large file from the RAID5 array to my Raptor disk, or extracted a file using WinRAR in the same manner.

Of course this is not the only problem, just an easy way to reproduce it. When I extracted large files I noticed that the Task Manager did not report memory being used under its graph, however the counter for available memory under ‘Physical Memory’ would go down a whole lot, using as much as 4GB of RAM when copying files at least that large!

This in itself is not an issue when memory is freely available as it’s a good idea to use it for cache. However something in the design of Windows causes it to be detrimental to performance. It wasn’t until I started using the Performance Monitor (perfmon) that I realized what the problem was and came upon a solution. When I monitored the system I noticed excessive Page Faults and Lazy Writes which would peak as performance dropped. So the reason for the slow performance is that when the file is cached in RAM it causes massive page faults, which also needs to be committed to the disk, which leads to disk trashing.

The issue gets more pronounced the larger the difference is between the disk that reads and writes, in the case of my system the RAID5 array has a read capacity of 160MB/sec, but the Raptor can only write at ca 70MB/sec. So therefore when reading a large file it takes only seconds to fill the cache. At which point the performance of my computer plummets, even browsing the start menu or opening the Control Panel takes ages, if I try to open the Add/Remove Programs applet which is filled with stuff it actually never opens until the copy process is completed!

After all this trouble looking for the solution I finally found the KB article that explains this issue:

You may experience a decrease in overall system performance when you are copying files that are larger than approximately 500 MB in Windows Server 2003 SP1 or in Windows Server 2003 SP2

Reading this article I was able to understand the issue more clearly, by default half of the system RAM is allocated for cache, so that was why I saw 4GB being allocated. Per the article I added the D-Word registry value SystemCacheDirtyPageThreshold under HKLM\System\CurrentControlSet\Control\SessionManager\MemoryManagement and set it to the smallest allowed value. Since I have 8GB RAM; 8192/64=128 (or 80 in HEX).

After a reboot the system felt just as responsive as Windows 2000 again when copying large files!

Since the issue intrigued me I did some more testing with WinDbg, I entered kernel debug mode and issued the command !defwrites. Here is the output using no modifications:

*** Cache Write Throttle Analysis ***

CcTotalDirtyPages:                    76 (     304 Kb)
 CcDirtyPageThreshold:            1048422 ( 4193688 Kb)
 MmAvailablePages:                1895872 ( 7583488 Kb)
 MmThrottleTop:                       250 (    1000 Kb)
 MmThrottleBottom:                     30 (     120 Kb)
 MmModifiedPageListHead.Total:        146 (     584 Kb)
Write throttles not engaged

As you can see under CcDirtyPageThreshold the system has 4GB allocated for cache and the message Write throttles not engaged telling you all is well, now see what happens when I copy a 50GB file from the RAID5 array to the Raptor disk:

*** Cache Write Throttle Analysis ***

CcTotalDirtyPages:               1048412 ( 4193648 Kb)
 CcDirtyPageThreshold:            1048422 ( 4193688 Kb)
 MmAvailablePages:                1274956 ( 5099824 Kb)
 MmThrottleTop:                       250 (    1000 Kb)
 MmThrottleBottom:                     30 (     120 Kb)
 MmModifiedPageListHead.Total:     575041 ( 2300164 Kb)
CcTotalDirtyPages within 64 (max charge) pages of the threshold, writes
  may be throttled

Check these thread(s): CcWriteBehind(LazyWriter)
Check critical workqueue for the lazy writer, !exqueue 16
Cc Deferred Write list: (CcDeferredWrites)
  File: fffffadf373e8f40 Event: fffffadf297d7848

Here you can see the system has now reached the limit for cache and has started throttling, at this point the file copy dialog box will actually say it’s completed and go away as soon as the whole file has been read into RAM, even though we still have 4GB of data in RAM to commit to disk. This is in part the reason why users were unhappy about the file copy process in Vista.

This is how it looks with a 128MB cache:

*** Cache Write Throttle Analysis ***
CcTotalDirtyPages:                    40 (     160 Kb)
 CcDirtyPageThreshold:              32768 (  131072 Kb)
 MmAvailablePages:                1783563 ( 7134252 Kb)
 MmThrottleTop:                       250 (    1000 Kb)
 MmThrottleBottom:                     30 (     120 Kb)
 MmModifiedPageListHead.Total:         10 (      40 Kb)
Write throttles not engaged

Now we only have a 128MB cache, of course we will still hit the throttle limit (much faster than before) but since it is only a 128MB cache we also have many less page faults, and therefore much less disk trashing.

Result is that the 1080p HD video plays with no lag, and I can actually bring up the Control Panel and Add/Remove Programs applet in a timely manner compared to Windows 2000!

There are some other applications that might be good to know about, for example if you came here looking for a solution to why you can’t make backups of your SQL databases or Exchange Server databases I can recommend the program eseutil.exe that can be found in the latest service pack for Exchange.

What this program does is copy files with direct I/O, not using the cache at all.

A similar program with a graphical interface is called Teracopy, I only tested that briefly but it seemed to pull the trick as well.

Be aware that copying files with direct I/O is only a good idea if you don’t intend to open the file afterwards, including the OS’ thumbnail preview or with antivirus scanners, else the file will just need to be read from disk once copied, instead of directly from RAM.

It should also be noted that if the file is large enough, and the system busy enough, for example a Terminal Server, a high profile IIS server, or just a really busy workstation then the system can actually fail the copy process entirely with a Delayed Write Fail error message and Event ID 50 in the Event Log.

To conclude I wanted to mention that I also tested this in Windows 7 Beta 1 and saw the same erratic behavior, however the registry key fix did not help, so if anyone knows how to fix this in Vista and Windows7 be sure to post a comment explaining how!

Written by Per Hansson

January 6th, 2009 at 4:42 pm