DSL Ideas and Suggestions :: Ultra-fast backup/restores



lzop is the *fastest* compression utility around. It compresses on average three times faster than gzip.

I've hacked backup/restore (again) to use lzop. If lzop is not installed, the hack defaults to using gzip.

The speed increase, especially when backing up, is substantial. Yet the resulting archives are only slightly larger.

Results (p3, device=hd):

Non-encrypted backup with gzip --> 42.04MB = 46.7 seconds
Same non-encrypted backup with lzop --> 46.72MB =5.8 seconds !!

Non-encrypted restore with gzip --> 5.8 seconds
Same non-encrypted restore with lzop --> 2.24 seconds !!

Encrypted backup with gzip --> 42MB = 54 seconds
Same encrypted backup with lzop --> 46.72 MB = 14.1seconds !!

Encrypted restore with gzip --> 14.2 seconds
Same encrypted restore with lzop --> 11.1 seconds !!

The hacked scripts are backwards-compatible with gzipped tarballs. This enables:

1. Painless user migration from plain or encrypted gzipped backups to lzopped backups - just place the
extensions (see below) on your mydsl partition and reboot.

2. Old copies of gzipped tarballs continue to be restorable.

3. Remastering to add the scripts to the base does not mean lzop must be added or always loaded. Without lzop, backup/restore should proceed using gzip.

To test:
~~~~~~

The altered scripts and lzop are inside extensions.

1. Download this
(r click, "save as")  and re-name fastbackup.dsl
md5sum 7d8996a88d83c1e9a706a849ef70d656 (please check)

also this and re-name lzop.dsl
md5sum a750aaf85deb4cd7041802c5d9bf17ef (please check)

2. Place both extensions on your mydsl partition and reboot including the mydsl={your_mydsl_partition}
and restore/protect options, as appropriate for your existing gzipped tarball. A normal restore using gzip should then occur.

3. When finished, hit 'backup' from the system menu or reboot.  Notice the increase in backup speed. Check your backup partition - you should find backup.lzo.des (if using 'protect'), or a new file backup.tar.lzo

4. On booting up again, the hacked script will try to find & restore one of backup.lzo.des, backup.gz.des, backup.des (old), backup.tar.lzo and backup.tar.gz (in that order).  If the candidate archive doesn't exist,  or there is no 'protect'  password or lzop where required, then the next candidate is found and restored if possible.

See the screen messages for details.

Unofficial scripts. No warranty, use AYOR, make copies of your tarballs beforehand, etc.

(Many thanks go to Robert and others who have encouraged me recently).
Quote (WDef @ Sep. 21 2005,08:03)
Non-encrypted backup with gzip --> 42.04MB = 46.7 seconds
Same non-encrypted backup with lzop --> 46.72MB =5.8 seconds !!

Non-encrypted restore with gzip --> 5.8 seconds
Same non-encrypted restore with lzop --> 2.24 seconds !!

Encrypted backup with gzip --> 42MB = 54 seconds
Same encrypted backup with lzop --> 46.72 MB = 14.1seconds !!

Looks like a lot bigger to me.

I'd rather wait the extra few seconds for a smaller backup. I don't have the space to store an extra meg or two.

-J.P.
Quote
..wait the extra few seconds ...


That's wait 41 extra seconds on that size tarball and a p3. It's an 8-fold speed increase in that example, for a ~10 per cent increase in tarball size.  I'd like some figures on a huge tarball.

People with tight backup headroom (and they're not everybody) don't have to load lzop.

I'd be grateful if people would test the work, which took some time and effort.

---------------------------------------------------------

BTW:  It might be safest for now to pass the backup device name directly through the boot options eg

restore=hdb6

or

protect=hdb6
Has anyone tried uzing "gzip" with a different compression setting?

I find that "gzip -2" compression level is a good compromise between speed and compression ratio.

I also expect the speed and compression ratio to be competitive with lzop, but I have not tested this.
Quote
I also expect the speed and compression ratio to be competitive with lzop, but I have not tested this.


There was a reasonably thorough survey published in Linux Journal a few months ago, using results over a number of different data types for a variety of different compression progs and levels. Log curves etc - more rigorous than this test on Windows, in which lzop was also the fastest.

gzip didn't come close speed-wise - lzop was the fastest of the lot of them.

There was another prog that achieved the very best compromise between speed and compression but it uses up gigantic resources - 900 MB ram etc

I'm talking about large speed increases without too unacceptable a drop in compression ratio.  A 10% increase in tarball size is not at all unacceptable to me.

Why not test lzop - can't hurt can it.
Next Page...
original here.