DSL Ideas and Suggestions :: Extensions can be recompressed with `advdef`
To all extension makers: I would like to bring a great program to your attention - it is `advdef` which is part of the AdvanceCOMP project. The source code can be downloaded here.
`advdef` recompresses zlib-compressed files with a 100%-zlib-compatible 7zip-originated 'Deflate' algorithm. The size of the reduced file is usually between 90-100% of the original. The size reductions are enough to suggest using `advdef` always. All '.dsl' and '.tar.gz' extensions are suitable. '.uci' extensions are not - my layman's explanation is that '.uci' files are iso images with compressed sectors rather than one whole compressed file. This does not make '.uci' inferior - `create_compressed_fs` already uses 7zip's LZMA in addition to 7zip's Deflate for optimal efficiency.
Point of interest: The latest version of `create_compressed_fs` (2.05) can create slightly smaller '.uci' files than the version in DSL (2.01) because it can choose between different compression levels for each block (zlib levels 1-9 or lzma) based on size - even though lzma is better in most cases, sometimes it loses due to higher overhead compared to zlib. This process is very time-consuming. It is used only if you specify `-L-2`. It is best-used during the final make. The source code for the latest version can be downloaded here. You might want to read the help for `create_compressed_fs`, because the syntax has completely changed between 2.01 and 2.05.
Quote
The size of the reduced file is usually between 90-100% of the original
This doesn't sound like much of an advantage, unless there is _no_ extra cost in processing (which also equals time) during decompression. Bzip2 can also create much smaller packages, and is already available in DSL, but is not used because of the dramatic increase in processing needed.
Quote
The latest version of `create_compressed_fs` (2.05) can create slightly smaller '.uci' files than the version in DSL
I recall reading something about the newer version of cloop being made specifically for kernel 2.6. Of course I could be completely wrong about this, and for all I know the cloop module version may have very little influence on the create_compressed_fs version. I do know that at one time I tried to install a recent version of cloop in another 2.4-kernel distro. The cloop module failed, but create_compressed_fs still worked. I thought it was odd. As with advdef, if the extra time needed extends to decompression in addition to compression, it would probably be faced with some resistance from some people. Otherwise, it sounds like something worth looking into.On a related nore: I tried without success to get anyone here interested in lzop a few years back. Much faster (eg 9X) than gzip or bz2 and only slightly lower compression ratio. A huge improvement for backup times.I think one of the major points is memory consumption... since gzip really takes up very little. This link may be interesting for this topic: http://tukaani.org/lzma/benchmarksReducing size is not of paramount interest when compared to decompression time and memory used. This is especially true given DSL's design philosphy of primarily being a nomadic live CD/frugal, where decompression of MyDSL extensions occurs many times and therefore time and memory required to perform such is paramount.Next Page...
original here.