hard disk compression

[Marin85]

Hi,

I have one 320 GB ultrabay HDD and one 1 TB external USB HDD and I am slowly but surely running out of space on both of them. So, I was wondering whether it is a good idea to let Windows perform HD compression on both of them to increase the free space. On these HDDs I mainly store backups of My Documents (.pdf, .djvu, .docx etc.), pictures, music, movies, so I was wondering whether the compression might have some (irreversible) negative effects on the quality of the stored files or even can cause data loss. In other words, are there any downsides to HDD compression beyond decreased performance? How about playing music (lossless .flac) or videos (1080p) from compressed HDDs?

Thanks in advance for any input,

Marin

[RealBlackStuff]

Time to invest (heavily) in a NAS with 4TB
http://reviews.cnet.com/external-hard-d ... 73127.html
http://www.engadget.com/2008/09/30/west ... duced-and/
http://www.engadget.com/2007/05/16/buff ... ed-to-4tb/

[BillMorrow]

i've had a 4 TB Drobo NAS for about 3 years..
works great..

[Marin85]

Guys, thanks for your replies! You just confirmed my fears that the best solution for me would be some external storage monster like the ones you mention Unfortunately, such systems are far from being mobile, which is a MUST in my case. So, as of now I am also considering some external enclosure plus WD Cavier Green 2TB HDD and my only hope will be that HDD capacities will be improving faster than my storage needs... A NAS with 4x of these would be even nicer though (Gosh, 1 PB should eventually do).

Anyways, from the little valuable information I could google out on this topic, it really seems it is pretty pointless to struggle with HDD compression when all these modern storage capabilities are available. Not to mention that, given the little space left on both HDDs, the compression algorithm won´t even want to start... But even if I manage to perform it somehow, it will be only a temporary solution...

[rkawakami]

Corollary of Parkinson's Law: "Data expands to fill all available storage" . It was true back in the days of 5MB hard drives and it's still true with today's 1TB drives.

[ArtShapiro]

Nobody has answered the actual question.

I have no inside knowledge of the compression algorithms used by Microsoft, but am totally positive that they are lossless. We use such volumes at work for our sophisticated mainframe emulation software, where a multi-gig Windows file acts as an entire diskpack for a Unisys mainframe. If one bit changed in this file, the world would come to an end as we know it. We have various databases in compressed volumes, and if the numeric data in these databases were to change, it would be detectable. It hasn't happened.

One cannot have lossy compression on critical data; imagine if your Quicken bank account changed a numeric value. It just ain't gonna happen.

Art

[rkawakami]

Hmm... yeah, I guess so . It appeared that Marin85 received the answer to the question(s) through the Google search that was mentioned. Hard drive compression is a "lossless" process, akin to taking all your files on the disk and putting them into a .ZIP file. Upon uncompressing, you get the same, identical file you started with. There is a throughput hit when reading/writing files from the disk as they first must go through the compression/decompression algorithm. It is quite a different process than say, MP3 encoding or compressing a digital image using the .JPG format. There you are throwing away bits that are deemed to be superfluous. It may sound and look like the original, but it's not. I know that there's not much benefit from trying to compress an already-compressed file; .JPG as an example. For other file formats like .flac, .mp4, .mpg and so on, I would expect them to act the same as they are already compressed to some extent.

[Marin85]

Corollary of Parkinson's Law: "Data expands to fill all available storage" . It was true back in the days of 5MB hard drives and it's still true with today's 1TB drives.

And how!

If one bit changed in this file, the world would come to an end as we know it.

Just wait another 2.75 years!

I sort of expected that HD compression would be supposedly a lossless process, at least in theory. Otherwise, M$ would not have included it in their newest OS, right... However, as it often turns out, things work in real life differently from what is advertised. So, I was afraid that the HDD compression might hide potential risks of data corruption or even loss. Art´s post made me again confident that my fears were unjustified. Apart from the performance penalty, there seems to be another downside though. In case sheep happens to happen, recovery of the lost data may be hard to impossible due to the compression. But then again, this probably applies to any other archive formats as well.

Before posting this, I didn´t actually realize that the option "compress ... to save space" is also available for single folders and files in Windows 7 (and probably Vista and XP too). And this utility has a great advantage over the classic archiving tools: it does not require a minimum of available free space to peform compression as it seems to be doing this natively on NTFS and thus does not need "swap space" like other archiving software. This is really great if you are like me, constantly running out of space..., because you can use it when you actually need it. And it is not quite the same as winzip since the latter can essentially create only a compressed copy of the original files and thus requires again some minimum amount of available free space (same goes for winrar and 7zip).

Anyway, for the sake of this thread I also did some small tests. As input I took a 4GB vmware image (virtual disk plus small configuration files). Here are the compression results:

WinRAR (best compression option selected) -> 1.45 GB
"compress ... to save space" -> 2.32 GB
WinZip -> (still waiting...) ->1.54 GB

A side note: Don´t try to cancel "decompression ... to waste space". In most cases it will take far more time than for the actual decompression to complete!


Cheers,

Marin

[mgo]

WinRAR (best compression option selected) -> 1.45 GB
"compress ... to save space" -> 2.32 GB
WinZip -> (still waiting...) ->1.54 GB

A side note: Don´t try to cancel "decompression ... to waste space". In most cases it will take far more time than for the actual decompression to complete!


Cheers,

Marin

An obvious and new use for compression would be with solid state hard drives. They are smaller, and surely can use some "real estate relief". My little tests with the native Windows 7 compression on an Intel SSD show some real world usefullness with compression on a cramped SSD. The compress/de-compress processes are exceedingly fast, and I can only assume there would be little or no lag time when reading the files.