dsl2unc

Forum: DSL Tips and Tricks
Topic: dsl2unc
started by: WDef

Posted by WDef on Sep. 14 2006,19:06

This evening I found myself making a unc extension from Robert's instructions and thought - why not a little script?

And 10 minutes or so later ... dsl2unc ! (With apologies to CBagger's deb2dsl).

Again, this is just an automated version of the instructions that Robert posted a while back, no creativity on my part required.

This seemed to convert the .dsl extensions I tried it on ok.

Code Sample

#!/bin/bash

# dsl2unc
# Just automates Robert's instructions for making a unc extension from a dsl.
# WDef Sept `06

help(){
cat <<"EOF"
Usage: dsl2unc /PATH/TO/EXTENSION
Creates a work dir in /home/dsl
New unc is output in /home/dsl
EOF
exit
}

#==================================================================

. /etc/init.d/dsl-functions # for ANSI colors

APP=${1}

if [ "$APP" = -h ] || [ "$APP" = --help ] || [ $# -ne 1 ];then help; fi
if [ ! -e "${APP}" ]; then echo "Can't find ${APP}"; exit 1; fi

if [ ${APP##*.} != dsl ]; then echo "${APP} is not a .dsl extension."; exit 1; fi

NAME=$(basename ${APP} .dsl)
WORK=${HOME}/$NAME

if [ -e "${WORK}" ]; then echo "${WORK} already exists."; exit 1; fi

mkdir ${WORK}

echo
echo "${WHITE}Making ${GREEN}$NAME.unc${WHITE} from ${YELLOW}$NAME.dsl ..${NORMAL}"
echo

cd ${WORK}
tar -zxvf ${APP}
find home tmp -type f 2>/dev/null | xargs tar -czvf user.tar.gz 2>/dev/null
rm -rf home tmp

cd ..

mkisofs -R -hide-rr-moved -cache-inodes -pad ${WORK} | create_compressed_fs - 65536 >$NAME.unc || exit 1
echo
echo "${GREEN}Finished. Now test your new ${YELLOW}$NAME.unc!${NORMAL}"

exit 0

Posted by WDef on Sep. 20 2006,11:48

Stony silence, ok, it's not rocket science, but it beats typing the same commands over and over again.

That 'find' line and 'rm' line following should include the opt directory if it's to work on all .dsl's, since a few have install-type scripts lurking in opt ie

Code Sample

find home opt tmp -type f 2>/dev/null | xargs tar -czvf user.tar.gz 2>/dev/null
rm -rf home opt tmp

Posted by mikshaw on Sep. 20 2006,12:20

Your work is appreciated =o)

Personally I don't have great interest in unc, as long as unc changes are not easily reversible...uci are more than adequate.

Your tar creation interests me, however. I've never really done anything with xargs, since i still don't understand how it works, but this method at least saves having to write a file list.

One thing I've wondered about, though -- not just in this script but in general -- if using "-type -f" might limit or even break some extensions. If an extension includes one or more symlinks, or even an empty directory, would those be excluded?

Posted by roberts on Sep. 20 2006,16:05

It is a shame that it seems that uncs and not that well understood.
As far as reversible let me address that...

Ucis are self contained and therefore easiy and dynamatically "loaded" and "unloaded".

dsls are memory hungry and "install" into the base (ramdisk) filesystem that has been made writeable. They, dsls cannot be "uninstalled". They can and do "uninstall" in a frugal/liveCD environment because they were in ram (ramdisk) so upon a shutdown/reboot they are gone.

Uncs are like ucis in that they are mounted under /opt. However they use unionfs to overlay the base filesystem. So you get extrememly low memory requirements to get the same benefits as a dsl.

Like dsls uncs are not self-contained and because they overlay the base filesystem many other processes could be using some or all of the overlayed resources. For example gtk2 and the latest gtk2 apps. Each is not self contained and as dsls they occupy much memory and on low memory machines they are often not possible to run. As uncs they can. But trying to "unload" gtk2 while they system is running a gtk2 app would be disaster. And on the other hand, to include gtk2 into each and every gtk2 apps would be much wasted space and download time.

Uncs also have the same benefit as ucis as upon shutdown/reboot their mounts go away, you always start with a pristine known system. A simple script could be used to "install" a unc onto a traditional hard drive install. But, I would wonder why it would even be necessasy or desireable.

Change is always resisted. Perhaps moving forward and with contributions such as dsl2unc, some of the many benefits of unc type extensions will become realized.

I known I am using cups.unc and XFree86.unc on a machine that could not otherwise use their dsl conterparts.

All DSL extensions use the same procedure to persist via the mydsl boot option and mydsl directory.

Posted by WDef on Sep. 20 2006,18:27

Quote

with contributions such as dsl2unc

One reason I posted this simple convenience is I know people enjoy using little tools (especially with ANSI colors, or pop-up boxes, I know I do). Rows of command line stuff is intimidating to newbies, so I'm hoping this will get people making some more uncs. I made 6 uncs effortlessly within minutes of finishing this. Psychologically, deb2dsl is more satisfying because the user has to type things into boxes (ie participates more => more ownership). Probably should add a few boxes ;=)

Uncs are great, a huge improvment on dsls (thanks to Robert). Ucis will probably (?) remain the "queen" of extensions - they don't require unionfs and they don't put anything irreversibly on the system filesystem.

But some things will never be ucis alone. An example is gnupg. I built a gnupg.uci that works (haven't got round to sending it to the repo). However, some applications that use gnupg-encrypted keys (an example is aespipe, another is loop-aes) expect to find gpg in one and only one place: /usr/bin/gpg. Whether or not gpg is in PATH anyway is irrelevant to these apps. I suspect there are other similar examples about. So there will remain a technical need to have gnupg.dsl or a gnupg.unc in addition to gnupg.uci. (You could run a wrapper to set up the symlink gpg -> /usr/bin/gpg, but this means running /etc/init.d/mkwriteable. Actually, the loop-aes author says you must physically *move* the gpg binary to /bin and then do a symlink /bin/gpg -> /usr/bin/gpg - who am I to argue with someone like Jari Russu?)

Quote

if using "-type -f" might limit or even break some extensions. If an extension includes one or more symlinks, or even an empty directory

Good point. Without that -type f, will get clobbering directories like /opt/bin passed into the unc (clang!). There's ways around that. I wrote an unrelated script (buried on my hd somewhere) with some bash that removed clobberers - I'll dig it up.

Posted by WDef on Sep. 21 2006,16:16

v0.2 - Testers wanted.

Now converts multiple dsls.
(Indenting seems to get lost when posting unfortunately).
==========================================

Code Sample

#!/bin/bash

# dsl2unc v0.2

# Just automates Robert's instructions for making a unc extension from a dsl.
# WDef Sept 2006

# Changelog v0.2

# Handles symlinks, opt and empty directories in user.tar.gz.
# Current working directory is assumed if path to argument extension is not given.
# Runs in batch mode: converts all argument .dsls to .uncs
#- eg to process somedirectory full of extensions, if dsl2unc is in PATH:
#- cd somedirectory; dsl2unc *
# Directory $DIR for $WORK and output is now settable.
# More lurid coloring :=)

#==========================// USER SETTING//===============================

# Put DIR on a hard drive partition if insufficient room in /ramdisk/home/dsl
DIR=/home/dsl

#==========================// FNS //========================================

help(){
cat <<"EOF"
dsl2unc v0.2 - convert .dsl extension to .unc
Usage: dsl2unc </path/to/ext1.dsl /path/to/ext2.dsl ..>
Creates a work dir in $DIR
New unc(s) is output in $DIR
Default DIR=/home/dsl
EOF
exit
}

dir_filter(){
# Filter out non-empty dir lines from stdin
while read LM; do
if [ -d "${WORK}/$LM" ]; then
if [ $(ls -1A ${WORK}/$LM| wc -l) -eq 0 ]; then
# empty dir - ok
echo ${LM}
fi
else
# files, symlinks - ok
echo ${LM}
fi
done
}

#======================================================================

. /etc/init.d/dsl-functions # for ANSI colors

results=/tmp/dsl2unc_results

rm -f ${results}

if [ "${1}" = -h ] || [ "${1}" = --help ] || [ $# -eq 0 ]; then help; fi

CURRENTD="${PWD}"
APP=""

for APP in $*; do

# If no path supplied, assume current working dir
[ $(echo ${APP} | tr '/' ' '| wc -w) -eq 1 ] && APP="${CURRENTD}/${APP}"

if [ ! -e "${APP}" ]; then
echo "${RED}Can't find ${APP}${NORMAL}" >>${results}
continue
fi

if [ ${APP##*.} != dsl ]; then echo "${RED}${APP} is not a .dsl extension.${NORMAL}" >>${results}; continue; fi

NAME=$(basename ${APP} .dsl)
WORK=${DIR}/$NAME

if [ -e "${WORK}" ]; then echo "${RED}${WORK} already exists.${NORMAL}" >>${results}; continue; fi

mkdir ${WORK}

echo
echo "${WHITE}Making ${GREEN}$NAME.unc${WHITE} from ${YELLOW}$NAME.dsl ..${NORMAL}"
echo

cd ${WORK}
tar -zxvf ${APP}
find home opt tmp 2>/dev/null | dir_filter | xargs tar -czvf user.tar.gz 2>/dev/null
rm -rf home opt tmp

cd ..

mkisofs -R -hide-rr-moved -cache-inodes -pad ${WORK} | create_compressed_fs - 65536 >$NAME.unc
if [ $? -ne 0 ]; then echo "${RED}Error making $NAME.unc" >>${results}; continue; fi
echo
echo "${GREEN}Now test your new ${YELLOW}$NAME.unc!${NORMAL}"

echo "${YELLOW}$NAME.dsl${WHITE} --> ${GREEN}$NAME.unc${NORMAL}" >>${results}
done

echo
echo "======================================================================"
echo
echo "${WHITE}DSL2UNC ${WHITE}RESULTS:${NORMAL}"
echo
cat ${results}
echo
echo "======================================================================"
echo "${GREEN}Finished.${NORMAL}"

exit 0

Posted by curaga on Mar. 23 2007,06:13

I had trouble with this (v2)
It had many errors and created a 2.6k unc from a 160k .dsl..
It also managed to !change! the md5sum of the .dsl!!

So improvement requested....

Posted by WDef on Mar. 23 2007,12:59

Pls provide some more information viz.
which extension were you trying to convert?

Posted by curaga on Mar. 23 2007,16:16

grub-splash. About half of the errors were from tar, but I don't remember them anymore..
Though it works when installing it..

The result unc only had one file, /sbin/grub-floppy (that's what read there) and the working dir also had only that..

Posted by WDef on Mar. 23 2007,19:17

I don't think that's posted yet.

Can you email it to me or upload it somewhere so I can attempt to replicate your issue?

Posted by ^thehatsrule^ on Mar. 23 2007,21:22

That's now posted in testing... < http://damnsmalllinux.org/cgi-bin....8;st=20 >

Posted by WDef on Mar. 24 2007,00:41

Curago, I don't seem to get any errors and your extension is converting to a unc ok.

I'm just doing

Code Sample

# ./dsl2unc grub-splash.dsl

as root with both script and extension in /home/dsl
(I use an older dsl while on the web so dsl2unc is in my home directory).

EDIT: Which reminds me I need to update this to try to preserve perms.

Posted by WDef on Mar. 24 2007,01:01

Ah-ha! Let me guess - you're running it something like

Code Sample

./dsl2unc somedir/grub-splash.dsl

That does go haywire. The problem is I got lazy on sorting out the relative paths. What's worse, I wrote a v3 that fixed this months ago and forgot about it. Mea culpa.

Try this:

===================================================

#!/bin/bash

# dsl2unc v0.3

# Just automates Robert's instructions for making a unc extension from a dsl.
# WDef Mar 2007

# Changelog v0.3

# Relative paths handled
# Added checks for work dir and filesystem
# Run as root to keep perms

# Changelog v0.2

# Handles symlinks, opt and empty directories in user.tar.gz.
# Current working directory is assumed if path to argument extension is not given.
# Runs in batch mode: converts all argument .dsls to .uncs
#- eg to process somedirectory full of extensions, if dsl2unc is in PATH:
#- cd somedirectory; dsl2unc *
# Directory $DIR for $WORK and output is now settable.
# Even more lurid coloring.

#==========================// USER SETTING//===============================

# Put DIR on a hard drive _linux_ partition if insufficient room in /ramdisk/home/dsl
DIR=/mnt/hda3/work

#==========================// FNS //========================================

help(){
cat <<"EOF"
dsl2unc v0.3 - make unc extensions from dsl extensions
Usage: dsl2unc <ext1.dsl ext2.dsl ..>
Creates a work dir in $DIR
New unc is output in $DIR
Default DIR=/home/dsl
EOF
exit
}

check_work_dir(){
# Check work directory looks ok
if [ -z "$DIR" ]; then "${RED}User setting work dir is unset.${NORMAL}"; exit 1; fi
case ${DIR} in
/mnt/*) D=$(echo ${DIR} | awk -F/ '{print "/" $2 "/" $3 }' )
FS=$(grep ${D} /proc/mounts | awk '{print $3}')
if [ -z "$FS" ]; then echo "${RED}$D not mounted.${NORMAL}"; exit 1; fi
case $FS in
ext2|ext3|reiser*);;
*fat*|ms*) echo "${RED}$DIR on $FS can't preserve perms - use linux filesystem.${NORMAL}"; exit 1;;
ntfs) echo "${RED}Writing to ntfs may not be a good idea ${NORMAL}"; exit 1 ;;
esac;;
/*|/ramdisk/*);;
*) echo "${RED}Err .. where exactly _is_ this $DIR anyway?"${NORMAL}; exit 1;;
esac
if [ ! -d "${DIR}" ] || [ ! -w "${DIR}" ]; then
echo "${RED}$DIR not a dir or not writeable.${NORMAL}"; exit 1
fi
}

dir_filter(){
# Filter out non-empty dir lines from stdin
while read LM; do
if [ -d "${WORK}/$LM" ]; then
if [ $(ls -1A ${WORK}/$LM| wc -l) -eq 0 ]; then
# empty dir - ok
echo ${LM}
fi
else
# files, symlinks - ok
echo ${LM}
fi
done
}

#===============================//MAIN //=======================================

. /etc/init.d/dsl-functions # for ANSI colors

if [ "${1}" = -h ] || [ "${1}" = --help ] || [ $# -eq 0 ]; then help; fi

check_work_dir

results=/tmp/dsl2unc_results

rm -f ${results}

CURRENTD="${PWD}"
APP=""

for APP in $*; do

# If no path supplied, assume current working dir

if echo ${APP} | grep -q -v '^/'; then APP="${CURRENTD}/${APP}"; fi

if [ ! -e "${APP}" ]; then
echo "${RED}Can't find ${APP}${NORMAL}" >>${results}
continue
fi

if [ ${APP##*.} != dsl ]; then echo "${RED}${APP} is not a .dsl extension.${NORMAL}" >>${results} ; continue; fi

NAME=$(basename ${APP} .dsl)
WORK=${DIR}/$NAME

if [ -e "${WORK}" ]; then echo "${RED}${WORK} already exists.${NORMAL}" >>${results} ; continue; fi

mkdir ${WORK}

echo
echo "${WHITE}Making ${GREEN}$NAME.unc${WHITE} from ${YELLOW}$NAME.dsl ..${NORMAL}"
echo

cd ${WORK}
tar -zxpvf ${APP}
find home opt tmp 2>/dev/null | dir_filter | xargs tar -czpvf user.tar.gz 2>/dev/null
rm -rf home opt tmp

cd ..

mkisofs -R -hide-rr-moved -cache-inodes -pad ${WORK} | create_compressed_fs - 65536 >$NAME.unc
if [ $? -ne 0 ]; then echo "${RED}Error making $NAME.unc" >>${results}; continue; fi
echo
echo "${GREEN}Now test your new ${YELLOW}$NAME.unc!${NORMAL}"

echo "${YELLOW}$NAME.dsl${WHITE} --> ${GREEN}$NAME.unc${NORMAL}" >>${results}
done

echo
echo "======================================================================"
echo
echo "${WHITE}DSL2UNC ${WHITE}RESULTS:${NORMAL}"
echo
cat ${results}
echo
echo "======================================================================"
echo "${GREEN}Finished.${NORMAL}"

exit 0

Posted by curaga on Mar. 24 2007,09:53

I also changed the working dir from /home/dsl to /temp/temp and ran it as root.. That directory did exist before trying it...

If you wish, just post the unc you made of it, I don't feel like messing with that for now.

But yea, I think I did something like that.. BTW, WDef, ever played Final Fantasy? It's curaga

EDIT: yea, but that was with v2

Posted by WDef on Mar. 24 2007,13:18

Not a big games person myself. I'll aim to get your nic right in future, typos notwithstanding.

I'm emailing you the unc so you can test it and post it.

EDIT: /temp/temp works.

Posted by torp on Mar. 28 2007,02:44

can someone instruct me how to use this script? Do I copy and paste the text above, and save it as a script? Sounds like a fantastic little tool.
guess i need to get off my ass and read up on scripting....

torp

Posted by andrewb on Mar. 28 2007,04:14

copy the script & save to a file called dsl2unc

type 'chmod 777 dsl2unc'

type 'dsl2unc <fname>.dsl'

It is a good idea to use the declobber script on the dsl file first - search for declobber in the forums. declobber is copied & saved the same way. To use it properly the system needs to be booted with the cheatcode 'dsl legacy' & then the declobber script run as 'sudo declobber <fmane>.dsl'. This script removes un-needed directories from the dsl files so they don't overwrite system directories.

Posted by Juanito on June 08 2007,08:49

There's something I don't quite understand here...

I am compiling in /opt/xorg72 with the following subdirectories:

/bin
/etc
/include
/lib
/share
/var

I ran dsl2unc and got a user.tar.gz that contained (I think - the file was too large to unpack in the limited memory I had) everything in /lib and a couple of the other subdirectories. Is this is what is meant to happen? By removing opt from the line:

find home opt tmp 2>/dev/null | dir_filter | xargs tar -czvf user.tar.gz 2>/dev/null

I can stop this but perhaps I have missed the logic?

Posted by Juanito on June 08 2007,12:33

Ah... If I make a unc extension from a dsl extension that unpacks to /opt/xorg72, I guess the unc will build on top of itself - is that the problem?

Posted by WDef on June 08 2007,21:02

Hi Juanito

If you've got it all compiled and running ok from /opt, you should build a uci instead of an unc

Uncs perhaps aren't best used to create a union with / of a filesystem mounted on /opt/whatever when that just effectively puts things in /opt/whatever - that's a redundant use of unionfs.

[I'll test this circumstance tomorrow - it's bed time where I am]

dsl2unc should just put the lot into user.tar.gz or so I thought ... otherwise you've found me a new bug.

Most .dsl extensions ordinarily have few files in /opt as things go ... usually wrappers and suchlike - if they're all in /opt they should just have tar.gz endings to avoid calling mkwriteable, and better still - they should be ucis.

Posted by Juanito on June 09 2007,05:06

Quote

If you've got it all compiled and running ok from /opt, you should build a uci instead of an unc

If only I could figure out how to make one...

Posted by WDef on June 09 2007,21:08

Guess we still have a doc problem.

This is from Saidin's notes in the old wiki and is about repacking a uci - I don't know if it's still there. You need to adapt it a bit to make a uci from scratch. In your case you're half way there since I don't think you need a user.tar.gz (no menu or icon).

Replace /opt/firefox in the following with /opt/xorg72.

Instead of steps 8 and 9, put any files and their directories that need to be written to writeable areas of the filesystem (/home/dsl, /opt (other than in xorg72), /tmp) such as menus or icons in the work/firefox/user directory, and make a list with eg:

cd user
find -type f >list

========================

To change/update a UCI file do the following: (example here is firefox.uci)
1. mount the uci as normal
2. mkdir work
3. cd work
4. cp -a /opt/firefox/.
5. cd firefox
6. mkdir user
7. cd user
8. tar -zxvf ../user.tar.gz
9. tar -ztf ../user.tar.gz > list
10. -- now edit the actual files as needed --
11. -- be sure to edit/update the "list"
12. tar -T list --numeric-owner --no-recursion -czvf ../user.tar.gz
13. cd ..
14. rm -rf user
15. cd ..
16. mkisofs -R -hide-rr-moved -cache-inodes -pad firefox/|create_compressed_fs - 65536 > ../firefox.uci
17. cd ..
20. rm -rf work
(Thanks to Robert Shingledecker)

Posted by Juanito on June 10 2007,05:52

Quote

Guess we still have a doc problem.

Oops, not really...I've been to the extensions page in the Wiki several times, but managed to miss the uci part - maybe because I was looking for unc at the time.

Posted by jpeters on June 10 2007,17:11

Quote (WDef @ June 09 2007,17:08)

You need to adapt it a bit to make a uci from scratch.

One thing that isn't included is the removal of the .xtdesktop icon when the uci is umounted. I'm not sure what the standard way of doing this is.

Posted by roberts on June 10 2007,17:15

Don't use the Status field in an Icon file for any extension and it should be automatically deleted upon shutdown.

Posted by jpeters on June 10 2007,17:35

Quote (roberts @ June 10 2007,13:15)

Don't use the Status field in an Icon file for any extension and it should be automatically deleted upon shutdown.

Yes, but uci's generally remove their icons when dismounted (without reboot). Also, for some reason the generated UCI fails if I move it anywhere other than where it was created. Thanks!