DSL Ideas and Suggestions :: Ultra-fast backup/restores

Quote (cbagger01 @ Sep. 21 2005,18:30)

Has anyone tried uzing "gzip" with a different compression setting?

I find that "gzip -2" compression level is a good compromise between speed and compression ratio.

I also expect the speed and compression ratio to be competitive with lzop, but I have not tested this.

I'd never heard of lzop until this post so I thought I'd try some comparisions of my own (including vs gzip -2). I got the following results

Code Sample

Format Compression Decompression Size

Code Sample

none - - 94.6MB
lzop 3.8s 2.0s 50.6MB
gzip -2 11.6s 2.3s 42.4MB
gzip 18.7s 1.9s 39.8MB
bzip2 83.8s 22.1s 37.5MB
lzma 458.4s 6.7s 30.7MB

This is on my 1.8GHz duron PC.

Clearly lzop offers much quicker compression times than even gzip -2. I can see how on a slow PC this could be beneficial.

This needs to be balanced with the draw backs : lzop is a uncommon format making it hard to manipulate the files in the backup from outside DSL. lzop offers lower compression ratios than gzip making it less useful when space is limited (e.g. on a pendrive).

Again, I'm not necessarily proposing that lzop should be added to the base.

It would be very easy to change the hack to be able to use lzma, bzip2 or any other comp programs as well as lzop and gzip, depending on what compression extension was loaded. You could then take your pick of max speed or max compression, or a compromise. How's that for a big picture.

ie This does not have to be solely about lzop in the longer term, so could we please not get bogged down in a discussion about the merits of various programs.

What I am asking for now are testers for the present extensions.

So far nobody has said they've tried these yet. How about some feedback?

-----------------------------------------
To respond to Friedgold's points:

Thanks for the figures. These will vary according to the mixture of data types of course.

Quote

..making it hard to manipulate the files in the backup from outside DSL.

If you've restored using lzop and decide to backup using gzip, just remove the lzop binary thus

Code Sample

sudo rm -f /bin/lzop

and hit backup. You should get a gzipped tarball back. Then either don't load lzop.dsl at next boot, or you can delete /move the previous lzopped tarball from the backup device. The altered script should then restore much as the original did. Of course a change of comp prog could easily be automated from (eg) the backup/restore widget.

Similarly, if you've restored ordinarily with gzip, you can load the two extensions and hit backup to get an lzopped tarball.

Lzop is available for Windows, and various *nixes etc. There's a statically-linked i386 linux binary anyway.

Quote

..l.ess useful when space is limited (e.g. on a pendrive).

Pendrives are very slow to write to, and getting rapidly bigger, so some who back up on pendrives might be looking for whatever speed gains they can get.

I just installed the 2 dsl's and tried it out. Works great and fast. my backup.tar.gz = 8.92 Mb and the same backup.tar.lzo = 9.74 Mb. Processor used to hit the roof when backing up (100%) now that's in the past. Backup time improved greatly. I run from a 512 Mb usb-stick (DSL 1.5) and i like the improvement in speed. (keeping this one!)

Thnx man!

All i need now is the right command-phrase in emelfm to "unpack in other panel"

I use emelfm a lot and need to see inside the backup.tar.xxx (or unpack it temporaraly)

Can you help?

'Configure' button --> Filetypes -->Add

Description Lzop tarballs
Extension tar.lzo, tzo

-->Add (under "Actions:}

Name Unpack in another panel
Command x cd %D; lzop -dc %d/%f | tar xvf -

--> OK
--> Done
--> OK

---------------------
Note it's lzop -dc, not lzop -c

BTW Patrick - Thanks for testing.

Next Page...
original here.