TechSpot

NTFS Compressed Volume File Corruption

By Nic
Feb 21, 2003
  1. Some months ago I bought a USB2 external hard drive to backup image files of my hard disks. One day I tried to restore a backup but found that the image was corrupt. Since then I always did a file compare after copying the files to the USB drive so that I could recopy any file that was corrupted during the copy. From experience it seemed that around 50% of files always needed to be recopied a second, or even a third time, before I got all files copied error free. I put the problem down to USB drives being unreliable as a backup solution.

    Only this week I decided to get a larger hard drive to use as a backup device and this time I fitted it into a drive caddy thus avoiding using USB. And guess what? I still get the same problems. The only thing that both drives had in common was the fact that I formatted them as Compressed NTFS Volumes because both were being used as backup devices, so it seemed like a good idea to maximize the available space.

    I have now decompressed both volumes and the file corruption no longer takes place. I should point out that I only ever experienced file corruption when writing to an ntfs compressed volume and not when reading from the volume. It seems to me that I may have discovered a bug in the way Windows XP writes to compressed volumes (I only use XP so I don't know if win2k suffers from the same problem).

    Or maybe the problem is unique to my hardware, but this seems unlikely, especially as it occurred on both IDE and USB devices.

    Has anyone else here used ntfs compressed volumes before?
    What was your experience?

    Any useful feedback would be most welcome.:grinthumb
     
  2. Phantasm66

    Phantasm66 TS Rookie Posts: 5,734   +7

    Could the compression have caused problems with your file comparison.....???? That would depend on how the file comparison worked. I wonder if you compared the files, but both were on compressed drives, if you would get the same error???

    What I am suggesting is that your means of comparing the files did not take the compression into account, and when it ran some checksum or whatever, it got a different result.

    However, you then say that if you attempted to copy the files a 2nd or 3rd time, then all was OK????


    Hm.... That's jolly interesting.

    I would certainly not use compression for a while and keep testing to see if you get no problems now on continually.

    This is an interesting scenario.
     
  3. Nic

    Nic TechSpot Paladin Topic Starter Posts: 1,549

    --------------------------------------------------------------------------------
    Phant ...
    Could the compression have caused problems with your file comparison.....????
    --------------------------------------------------------------------------------

    I use 'Windows Commander' to do both copying and file compare.

    The file compare works consistently. If I re-run a file compare on a set of files it is always the same files that show up as corrupt the second time also. When I have successfully copied all files, then the file compare always shows the files as OK regardless of how many times I run it. The fact that the files are on a compressed volume is transparent to the OS and any programs that access the files.

    Heres some info ...

    NTFS Compressed Files & Folders

    Its a strange problem and probably hasn't been discovered yet because not many people are using ntfs compressed volumes and doing file compares. It had me very frustrated for a long time. One other thing - you need to copy several large files to notice this problem as it only shows up occassionally. Copying several large (e.g. 700MB) files virtually guarantees that some of your files will be corrupted.:blush:
     
  4. Phantasm66

    Phantasm66 TS Rookie Posts: 5,734   +7

    Right enough I have just ran an md5sum on two files, both of which were identical, except that I compressed one with NTFS compression.

    They got exactly the same MD5 sum result.
     
  5. Phantasm66

    Phantasm66 TS Rookie Posts: 5,734   +7

    Can you recreate the original environment in which the problem happened, and then compare an original file and a backup file with this tool, which you must run in a command prompt:

    http://phantazmm.freeservers.com/md5sum.exe
     
  6. Nic

    Nic TechSpot Paladin Topic Starter Posts: 1,549

    --------------------------------------------------------------------------------
    Phant ...
    Can you recreate the original environment in which the problem happened,
    and then compare an original file and a backup file with this tool ...

    ---------------------------------------------------------------------------------

    Sure, here are my results ...

    1.
    Copy to *ntfs compressed* folder on drive.
    ------------------------------------------
    md5sum = 5e034d34239e127caaf44f38adbbcd5e - original file [good]
    md5sum = 33c4a9263abc18f0c5964c44d63a2dae - copied file [5 attempts still bad]

    2.
    Next I decompressed the folder with the corrupted file and re-run the compare.
    The result was unchanged [error still exists].

    3.
    Re-Copy the file to the now *uncompressed* folder.
    --------------------------------------------------
    md5sum = 5e034d34239e127caaf44f38adbbcd5e - original file [good]
    md5sum = 5e034d34239e127caaf44f38adbbcd5e - copied file [first attempt]

    [the file I used (733MB) Mandrake.LINUX.V9.0.Powerpack.Edition.CD6.ISO]

    Strangely, I was unable to copy the file to the compressed volume even after five attempts when I gave up. Normaly the situation isn't quite this bad and eventually get a successful copy. Maybe some files (i.e. with particular bit patterns) are more affected than others.

    Its easy to test this problem by creating a temporary folder on your hard disk, right-click to select properties, click the advanced button, then select the compression option.

    Very strange indeed!:confused:
     
  7. Phantasm66

    Phantasm66 TS Rookie Posts: 5,734   +7

    Man, that's a bit crazy.

    There's got to be a good, logical explanation for this.

    In my example, both files where on an uncompressed volume. I merely compressed a single file.

    I wonder if its possible to have a corrupted compression engine, if that's what you call it...

    I take it chkdsk on any drive with nothing compressed is fine? What if you format a partition with NTFS, compress it all, copy some data on and then run a chkdsk? What happens then?

    Did you install any compression software? Have you tried REINSTALLING any service packs?
     
  8. Nic

    Nic TechSpot Paladin Topic Starter Posts: 1,549

    ---------------------------------------------------------------------------------
    Phant ...
    I wonder if its possible to have a corrupted compression engine
    ---------------------------------------------------------------------------------

    I've been having this problem for about a year now.
    During that time I have reinstalled, and updated, my windows system many times. I have even run the 'sfc.exe /scannow' command to fix any suspect system files, but the problem still exists. Originally I had compression on the full drive volume. I only now tried it on a single folder today (because I decompressed my drive yesterday) just to run some tests as you suggested. Furthermore, the corruption is not consistent, I do manage to copy most files without error.

    I don't suppose you could try creating a compressed folder on your drive and giving it a bash?:grinthumb

    Enough said, I won't be using ntfs compression again!:dead::cool:
     
  9. Phantasm66

    Phantasm66 TS Rookie Posts: 5,734   +7

    But it SHOULD work! And if you have reinstalled, then that's pretty bad.

    Either its a hardware related issue, or you keep installing something that's causing this problem, without realising it... Do you know if you get the trouble with a fresh, clean installation, with NO other software??
     
  10. Nic

    Nic TechSpot Paladin Topic Starter Posts: 1,549

    -----------------------------------------------------------------------------
    Phant ...
    Do you know if you get the trouble with a fresh, clean installation, with NO other software??
    -----------------------------------------------------------------------------

    I can't really verify that without rebuilding my current system.
    I'm not running anything unusual, just the normal stuff, such as Easy CD Creator, MS Office, McAfee Antivirus, etc.
    My setup and installed software has varied over the last year, but the problem has been persistent. Besides, whats the point of using compression if it can be rendered unusable when certain software is installed?:confused:

    In order to eliminate the possibility of my setup being the root cause, I really need to get someone else to give it a shot. Really it only takes about 10 mins to try it out.:grinthumb
     
  11. Phantasm66

    Phantasm66 TS Rookie Posts: 5,734   +7

    I was thinking maybe not by design but by mistake some software you have added has rendered the ability to compress data on an NTFS partition corrupt.

    Post exactly what you would like someone to try, step by step.
     
  12. Nic

    Nic TechSpot Paladin Topic Starter Posts: 1,549

    --------------------------------------------------------------------------------
    Phant ...
    Post exactly what you would like someone to try, step by step.
    --------------------------------------------------------------------------------

    Step 1 - Create a new NTFS compressed Folder.
    1. Create a new folder on any NTFS volume.
    2. Right-Click on the new folder and select 'properties'.
    3. Click the 'Advanced' button.
    4. Tick the box labelled 'compress contents to save disk space'.
    5. Click 'Apply'.

    The new folder will now have an attribute of 'c' showing that compression has been enabled.

    Step 2 - Test the file copying to see if corruption takes place.
    1. Copy any large file (e.g. 700MB) to your new compressed folder.
    2. Use a tool such as 'md5sum.exe' (see previous post for link) to compare the copied file with the original.
    3. Delete your compressed folder.

    TEST COMPLETE :eek:

    If you don't get an error the first time you can try and copy the same file again. I only get file corruption on average about 50% of the time, but as I said previously, its random.:cool:
     
  13. Phantasm66

    Phantasm66 TS Rookie Posts: 5,734   +7

    Great. I knew what to do, but some other readers might not, and might want to try it too.

    I'm also stoned so having step by step instructions is good for me, too.... ;)

    I will try in a spare moment and post back.
     
  14. Phantasm66

    Phantasm66 TS Rookie Posts: 5,734   +7

    I just did your test using an 800 MB mpeg file.

    The two md5sum results were IDENTICAL.

    Have you installed the latest drivers for your motherboard???
     
  15. Nic

    Nic TechSpot Paladin Topic Starter Posts: 1,549

    ---------------------------------------------------------------------------------
    Phant ...
    Have you installed the latest drivers for your motherboard???
    ---------------------------------------------------------------------------------

    Yes.
    My motherboard has a via chipset, and I'm using the latest via 'Hyperion 4.45' drivers.

    It could be a hardware issue, but as I previously said I don't get corruption all of the time.

    Most times the files I am copying are either ISO image files or Hard disk image files, though I have had corruption with large AVI files also. I don't yet have enough data to see if corruption is related to the particular bit pattern of some files, i.e. some files might be easily copied without corruption whilst others are prone to errors. I say this because whenever I have to recopy a file that has been corrupted, it often needs to be done several times before I get a good copy.

    But, just as you mentioned previously, it could just be my hardware, or a software driver issue. I guess I won't really know for sure until I replace my current PC sometime this year.

    Did you try the test a few times just to be sure you weren't just lucky?

    Thanks for giving the test a go Phant, its nice to know that someone is always willing to help. cheers! :grinthumb
     
  16. johannes73

    johannes73 TS Rookie

    If you have got this PC yet:
    Have you tried it out with McAfee having deactivated?

    I have heard about the same problem, but NOT on a compressed volume, but using a Athlon XP and McAfee Antivirus applying some errors and differences on a MDB(Access2K)-File.
     
  17. Nic

    Nic TechSpot Paladin Topic Starter Posts: 1,549

    I found the source of my error several months ago. My hard drive (external usb2) had some defective sectors and and the circuit board (on the drive) was making poor contact with the connectors on the drive body. For some reason, using compression made any errors more apparent, probably due to error correction not being as effective on a compressed volume (just a guess). I fixed the drive and I'm not using compression anymore, so it is no longer a problem. Thanks anyway. :)
     
Topic Status:
Not open for further replies.

Similar Topics

Add New Comment

You need to be a member to leave a comment. Join thousands of tech enthusiasts and participate.
TechSpot Account You may also...