René B. Færch

HDD loosing data over time... is it normal?

Recommended Posts

Hi

I have a few questions regarding backups. My situation is that I have a laptop and an external harddisk that I use for backups. Normally I copy my photos, music, and software directly to the external harddisk and zip my documents into one file (containing documents, MS Outlook archive, etc on around 2 GB) to save time and keeping the amount of files down.

When I'm not backing up or restoring files on my external harddisk it is disconnected from my computer and stored in a closet.

Only very few times I have been needing to access my files on the drive and then I have experienced that a few files has been are corrupted in some way and cannot be read anymore! I have experienced this both with my previous and current external harddisk (one was an IBM and the other one a Western Digital).

My questions:

1) Is it normal that data on a harddisk drive gets corrupted by time?

2) Do accessing the files on a harddisk "renew" the data in a way that keeps it valid for a longer period?

3) What way of backing up data is the most reliable - HDD/DVD/CD/tape/..?

4) What way of backing up data is the most reliable and not too expensive for personal use?

5) What is the average survival time for data on a HDD, DVD and CD? Do the DVD or CD surface layer make any difference on this?

Hope someone can help me answer some of these questions :-)

René

Share this post


Link to post
Share on other sites

Some thoughts:

2) You would need to move the data physically to another location to have it magnetically rewritten to disk. Move it to another hard disk and then back would do. Or making a copy and then deleting the old (after VERIFYING the copy), I assume. I have never heard anybody doing anything like this. Just backup. If you need to monitor the data integrity, then you need some DAM application with CRC checks or something. Or a big IT department.

3) Optical media can degrade due to fingerprints, direct sunlight and many other things. It can crack if subjected to pressure (think: books and stuff). A successfull burn could be full of errors that the built-in error checking is able to fix. After a few months of media degradation though, the media could be full of errors. Media and burn quality matters. CD has best error checking. Keep them in special cd sleeves in a folder. On the other hand, it is SO easy to drop a hard drive... The most reliable way to backup is to have a backup schedule with multiple generations of backup stored off-site. You could schedule weekly full backups and daily incremental backups to a network attached hard drive (they don't cost that much anymore). Then you could change that drive once a week the morning after the full backup. When the lightning finally strikes your house and destroys both the computer and the backup drive and the house, you have the other backup drive somewhere else. Maybe you also put it in a faraday cage, would there be an unexpected EMP and protect it from floods. :D Use surge protectors at least.

4) affordable backup is external hard disk (here it costs ~1 euro per 10 GB). Optical media is not much cheaper. Having a network drive lets you keep the backup drive somewhere else than your disk. A healthy thing to do, probably. Probably worth the slight increase in price (think about these unattended scheduled backups).

5) Usually the survival time of a hard disk (at least for me) is defined by the "disk full" feature. Western digital Raptor/Velociraptor/Caviar have good mean-time-to-failure values. Maybe others too ;) "Mean time" does mean that YOUR drive probably dies much faster :D but it's a good measure of reliability. I don't know the difference in cd quality. It exists, I know.

Share this post


Link to post
Share on other sites

Hi ralstra and thank you for your quick reply :-)

Regarding your replies:

2) Without knowing, I figured that when the I/O system access data on a disk and find an correctable error it would correct the data instead of waiting until so many errors exist that the data cannot be corrected.

I haven't heard about DAM applications before. What can it do besides telling me when my data is broken? Do you know about any 'famous' applications?

3) Considering different medias for storage what do you think about USB memory sticks - less safe than HDD/DVD/CD? Do bigger companies still use magnetic tapes for storage and is it considered to be safer than other medias?

4) Is network backup/off-site backup something you would recommend me to put up myself or subscribe on a hosting like hosting of a web domain?

Thank you :-)

Share this post


Link to post
Share on other sites
Hi ralstra and thank you for your quick reply :-)

Regarding your replies:

2) Without knowing, I figured that when the I/O system access data on a disk and find an correctable error it would correct the data instead of waiting until so many errors exist that the data cannot be corrected.

If there is a file system error, running chkdsk/r should be able to detect and fix it. If windows encounters a file system error, it will mark the whole disk volume as 'dirty' and somethings like defrag will not be possible until you fix that error. But you still have to run chkdsk manually. This is my understanding, and I could be wrong (i'd gladly welcome a correction in that case).

For possible hardware faults, you could run the disk diagnostic utility from the manufacturer to scan the disk for errors. It ought to be freely available from the manufacturer's website.

As for backing up, I am quite paranoid. I backup important data regularly to a secondary SATA internal disk, a USB Freeagent drive and also burn it to optical media. Some files are also backed up to online storage. The optical media and external drive, i keep in my lab for physical separation from the home computer in case of a fire, damage etc to the premises.

Share this post


Link to post
Share on other sites

Research "QuickPar" and PAR2 files. You run the utility once, over your source data, and it spits out a collection of PAR2 files, that are like checksum files, but they also contain reconstruction data. Then you store those PAR2 files with the original data files. When it comes time to read back the data files, if they have bad sectors or something, then you can use the PAR2 files to repair the defects and reconstruct the original data. You need to have PAR2 files of sizes equal or greater than the size of the damaged part of the files. The size and number of PAR2 file chunks is configurable when you generate the PAR2 files.

Also, if you are encountering file corruption, then test out your RAM with a bootable Memtest86+ CD.

Share this post


Link to post
Share on other sites

I contacted some manufacturers of harddrives as well as this forum and today I received these answers from Western Digital Customer Service and Support:

1) Is it normal that data on a harddisk drive gets corrupted by time?

Time alone will not corrupt data on a hard drive. As long as the hard drive hardware has no issues the information will be retained. Outside forces such as the operating system, not safely removing the drive, physical damage will all possibly cause corruption. In most cases (other than hardware) the operating system will somehow cause corruption.

2) Do accessing the files on a harddisk "renew" the data in a way that keeps it valid for a longer period?

Accessing the files will not keep then valid for any longer than not accessing them.

3) What way of backing up data is the most reliable - HDD/DVD/CD/tape/..?

All of the methods (HDD/DVD/CD/Tape) you mention are reliable as backups but all with limitations. HDD backups can be prone to corruption etc and can physically be damaged from drops/falls etc. DVD/CD media does physically degrade over time but in most cases is read only so it does not get effected by corruption etc. Tape backups also can physically degrade and in most cases would not become corrupt.

4) What way of backing up data is the most reliable and not too expensive for personal use?

In most cases HDD backup is the most reliable backup type for the price. Of course you always want to have at least copy of your information so in most cases if the external drive fails you still have a copy internally etc.

5) What is the average survival time for data on a HDD, DVD and CD? Do the DVD or CD surface layer make any difference on this?

I do not have any sort of "survival" time for data stored on a hard drive. Western Digital hard drives have a less than 1% failure rate over 5 years so you should be able to count on 5 or so years without issues.

Share this post


Link to post
Share on other sites

And here's the reply from IBM's developerWorks forum on the topic. Due to this guy tape is considered the most reliable media for backup:

1) Is it normal that data on a harddisk drive gets corrupted by time?

No, it's not normal at all. It's supposed that information stored on HDD should not get corrupted unless HDD is exposed to some kind of radiation or humidity.

2) Do accessing the files on a harddisk "renew" the data in a way that keeps it valid for a longer period?

Hmmm, no. Accessing data from time to time is an old trick used when storing data on tape due to some chemical characteristics magnetic tapes has that make their layers stick together. When using HDD it's supposed this anomalies does not happen.

3) What way of backing up data is the most reliable - HDD/DVD/CD/tape/..?

Tape.

4) What way of backing up data is the most reliable and not too expensive for personal use?

Depending on you current platform. I have seen people whose computers run Linux using Backzilla. Using rsync is a wise option, you can set a backup server and start client replications to that backup server and then run a script that copies all your data to an external media.

5) What is the average survival time for data on a HDD, DVD and CD? Do the DVD or CD surface layer make any difference on this?

This has to do with a number called MTBF (Mean Time between Failures)

Share this post


Link to post
Share on other sites
My questions:

1) Is it normal that data on a harddisk drive gets corrupted by time?

2) Do accessing the files on a harddisk "renew" the data in a way that keeps it valid for a longer period?

3) What way of backing up data is the most reliable - HDD/DVD/CD/tape/..?

4) What way of backing up data is the most reliable and not too expensive for personal use?

5) What is the average survival time for data on a HDD, DVD and CD? Do the DVD or CD surface layer make any difference on this?

Hope someone can help me answer some of these questions :-)

René

1) No, but it is not unheard of either. Disks fail slowly when exposed to heat, vibration, humidity, radiation, etc. Small amounts can cause small failures on a disk over time - large amounts can cause total disk failures.

2) NO

3) High-performance (NOT consumer-grade tape backup) tape stored in a climate controlled vault, followed closely by a RAID 5 array of disks stored in the same vault.

4) Right now a RAID 5 server appliance, or even RAID 1 mirrored disks in a server appliance, seems the best "bang for buck". There are a number of external enclosures, some directly attachable, some networkable, that provide RAID 5 or 1 in a simple box. There is also the Drobo, which provides it's own RAID management and security transparently, but frankly I have no idea how reliable it is over a pure RAID 5 array - but it is supposedly easier to admin and expand. Personally, I would just suggest an array of two 1 TB disks in a single enclosure with RAID 1.

5) Years and years, or just minutes. I just retired two drives that were 4 years old that never lost any data, and their replacement 1TB drive failed the first day (thank god I hadn't moved data to it yet!). That's what backups are for, and why really important data (personal photos, papers, etc.) should be backed up in several ways. I keep making DVDs of backups of My Documents and other files that I create, despite having a HD-based backup system....

6) The question you DIDN'T ask: it's not just about the media, it's where you store it. Having a totally redundant RAID 5 or RAID1 array, or commercial-grade tape backup, is nearly worthless if all the copies sit in your house and it burns down to the ground. Or floods. Or is burgled. Those few DVDs of backup data you left at your family members (in a sealed envelope if need be) may be the wisest investment you can make...

Share this post


Link to post
Share on other sites
Thanks :-)

What online storage do you use?

The files are backed up to my lab/department server. I take the external drive to the lab and give it to the IT guy who does it for me. :)

Share this post


Link to post
Share on other sites

BTW, heat and humidity are the enemies of magnetic thin films on the drive platter.

But we are talking about temperatures higher than normal operating temps (>100 deg C) that would cause disruption of the magnetic moments of the magnetic domains in the thin films.

Normal operation is more likely to mess up the mechanicals of the drive than the magnetic media itself. Infact, I'd speculate that most of the common drive problems arise from mechanical (spindle/actuator arm) or electronic (circuits/controller) causes than from degradation of magnetic media.

Humidity can likely cause localized corrosion and damage the thin film.

Share this post


Link to post
Share on other sites

Thank you :-)

4) Right now a RAID 5 server appliance, or even RAID 1 mirrored disks in a server appliance, seems the best "bang for buck". There are a number of external enclosures, some directly attachable, some networkable, that provide RAID 5 or 1 in a simple box. There is also the Drobo, which provides it's own RAID management and security transparently, but frankly I have no idea how reliable it is over a pure RAID 5 array - but it is supposedly easier to admin and expand. Personally, I would just suggest an array of two 1 TB disks in a single enclosure with RAID 1.

The array of two disk in a single enclosure, can you recommend any specific?

Does it has it's own RAID controller and if it can be connected using FireWire or USB? I'm thinking of connecting it to my laptop.

René

Share this post


Link to post
Share on other sites
I contacted some manufacturers of harddrives as well as this forum and today I received these answers from Western Digital Customer Service and Support:
1) Is it normal that data on a harddisk drive gets corrupted by time?

Time alone will not corrupt data on a hard drive. As long as the hard drive hardware has no issues the information will be retained. Outside forces such as the operating system, not safely removing the drive, physical damage will all possibly cause corruption. In most cases (other than hardware) the operating system will somehow cause corruption.

This is NOT accurate.

I have first-hand experience... In fact i was going to create a new topic with my own findings.

DEMAGNETISATION / SUPERPARAMAGNETISATIONS / THERMAL DECAY is what you want to search for.

DATA GETS CORRUPTED ON PERFECTLY FINE DRIVES IF NOT ACCESSED OVER PROLONGED PERIODS OF TIME (2+ years in my case). Hard drives (high density) are just NOT OK for offline archival.

Share this post


Link to post
Share on other sites
This is NOT accurate.

I have first-hand experience... In fact i was going to create a new topic with my own findings.

DEMAGNETISATION / SUPERPARAMAGNETISATIONS / THERMAL DECAY is what you want to search for.

DATA GETS CORRUPTED ON PERFECTLY FINE DRIVES IF NOT ACCESSED OVER PROLONGED PERIODS OF TIME (2+ years in my case). Hard drives (high density) are just NOT OK for offline archival.

Interesting. Do you know if accessing the data will prevent it from becoming corrupted or do you need to rewrite the data?

Also, do you know if there are any medias more reliable for doing backups?

Share this post


Link to post
Share on other sites
This is NOT accurate.

I have first-hand experience... In fact i was going to create a new topic with my own findings.

DEMAGNETISATION / SUPERPARAMAGNETISATIONS / THERMAL DECAY is what you want to search for.

DATA GETS CORRUPTED ON PERFECTLY FINE DRIVES IF NOT ACCESSED OVER PROLONGED PERIODS OF TIME (2+ years in my case). Hard drives (high density) are just NOT OK for offline archival.

Interesting. Do you know if accessing the data will prevent it from becoming corrupted or do you need to rewrite the data?

Also, do you know if there are any medias more reliable for doing backups?

Rotate the drives (rewrite). Keep backup of backups. Otherwise, i was to lose 2 TB of data (10+ rotten drives). Yes, hard drives rot like meat!

Share this post


Link to post
Share on other sites
Hi

My questions:

1) Is it normal that data on a harddisk drive gets corrupted by time?

2) Do accessing the files on a harddisk "renew" the data in a way that keeps it valid for a longer period?

3) What way of backing up data is the most reliable - HDD/DVD/CD/tape/..?

4) What way of backing up data is the most reliable and not too expensive for personal use?

5) What is the average survival time for data on a HDD, DVD and CD? Do the DVD or CD surface layer make any difference on this?

Hope someone can help me answer some of these questions :-)

René

Hi!

1) It is not normal but it is possible. Every HDD has a parameter called "Error rate (non-recoverable)" and it is usually 1 bit in 10^14 bits read; 10^15 for newer and enterprise drives; 10^16 for server-grade drives. This doesn't seem so much, and it isn't, but if you make great use of your HDD, something of your data WILL be corrupted.

If this happens to a .jpeg, you could not notice any difference, but if the file is an executable with CRC check, it is lost.

Another coulprit is the RAM, without ECC support, you are going to have, from time to time, a change in the data stored, where a 0 becomes 1, and the file becomes corrupted. If you were going to write this file to the disk, you are also going to have another corrupted file.

There are also other cases, like the infamous NFORCE4 ATA drivers or the first AMD Phenoms TLB bug, that adds to the count.

This is called "Silent Corruption" (because you won't have any alert about it) - and not everyone is aware of it.

2) No.

But doing so, you could notice a problem sooner than later, and then let the HD handle for you. It could be a checkdisk, or a sector reallocation, either way, it could save your data integrity.

Since HD are electromechanical devices, being used it also does good for their health.

So, you should make a full HD reading from time to time (never store your HD in your closet for months).

3) Let others handle it for you. Seriously, you give the data to networks, corporations, online services, friends (!), and the problem of data integrity is now theirs.

Second bet is tape, but it is also very impratical to use in a non-corporate envinroment...

4) DVD is.

- DVD+R have slight better error correction and electromagnetic immunity than DVD-R, so use them instead

- UDF have better file handling than ISO/JOLIET (go for v.2.01, it is the latest XP compatible)

- DVDs should never be exposed to direct sunlight (or whatever IR emitter), and should be handled with care

- Label them with the year, and burn them again every 3-5 years - they could last 10-20 years, but it's better not play with the luck

4) HDD, too.

Since you can't backup 500GB of data with only DVDs!

Just keep in mind RAID 1 is not a backup solution. If you have a corrupted file, RAID 1 will copy it to the other disk - this is not very SMART.

RAID 5 is slighty different, it also could save you from the "unrecoverable bit error" - but only if using a very costly hardware controller. You have also to pay attention to the RAID write hole.

So it is advisable to use a UPS - in my opinion, is not advisable to play with RAID disks at all.

Linux literate should use ZFS of Solaris (FreeBSD).

We should use a simple spare disk with a backup program, es. SynckBackSE.

You set up the program to do a fully automated backup of your hard drive data to another hard drive (every 1/7/15 days, you decide), and that's all.

5) I'll try to give you an answer, but nobody know the exact numbers since there are too many variables!

There is a difference in CD/DVD surface, as in media handling - CD are more secure. At their time, they were stated to last even 100 years - and I have in fact 12+ years disks that I can read without errors.

But the same could be said for my HDDs, magnetic tapes, or, to a lesser extend (they suck) floppy disks.

I'd say CDs could store securely your data for 7-8 years, while DVDs and HDDs for 5-6 years.

There is also a great difference in HD surfaces, and you can read about it right from this same site:

http://www.storagereview.com/guide/mediaMedia.html

I'd like to add that new perpendicular HDDs use a slighty lower reliability substrate...

So, my recipe for data preservation?

- Do not overclock

- Use high-quality DVD/HDDs (and components, for what it matters)

- Use a Journaled FS

- Use a UPS

- Backup your data to another disk (DNAS, NAS, SAN, JBOD)

- Double Backup your personal data to DVD+R

- Change your HDDs over time - a nice excuse for upgrade

- Use ECC memory

Share this post


Link to post
Share on other sites

Hi Mike and thank you for your very thorough answer on my questions :)

Speaking with a guy in the IBM DeveloperWorks forum asking the question "Do accessing the files on a harddisk "renew" the data in a way that keeps it valid for a longer period?"

He replied "Hmmm, no. Accessing data from time to time is an old trick used when storing data on tape due to some chemical characteristics magnetic tapes has that make their layers stick together. When using HDD it's supposed this anomalies does not happen."

I then asked "Without knowing, I figured that when the I/O system access data on a disk and find an correctable error it would correct the data instead of waiting until so many errors exist that the data cannot be corrected."

He replied "Yes, exactly.".

Would you agree or disagree with this?

René

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now