Background (not for very lazy people )
During our last and first official team meeting a point has been made that packages should be built with virtual machines. This is to ensure a clean environment, allow cross-package and testing in a basic and clean system. In a optimal scenario one would use snapshots to start clean every time.
That's actually a really good point although I'm not a big fan of VMs for packaging.
Why
- Speed: VMs are slow(er) (at least for packaging and even more for some other special uses).
- Workload: Managing VMs and especially snapshots with them is a lot of work. And you can loose track with all that stuff like hard disks, configurations, snapshots, ...
- Storage: You need many hard disk and need to be able to create new ones on the fly (at least for snapshotting). Those hard disk are not necessarily very space efficient
- Isolation: VMs are almost a completely separate system, so without additional setup (--> workload) it can easily get quite painful to work with.
Alternative: Chroots (not for lazy people )
So...? If it's that bad you gotta have something else? Yes! Chroots are an alternative, which I've been using a while now for packaging 32-Bit on a 64-Bit system. It works pretty well.
Some cons for chroots:
- Isolation: The lower level of isolation doesn't require and allow you to boot a system from scratch. You're just using different binaries. Although this saves you booting and has other advantages, it could rarely cause problems for some very special packages. But as it's a less complete system, you usually notice that you need to switch to VMs or real hardware because something fails. I.e. usually you can say for most packages if it works in the chroot, it works on real hardware/VMs too.
- Flexibility: You can usually only package for the same arch, you're physically running. x86_64 is an exception as x86 code can run on a x86_64 cpu. As I'm only packaging for x86* that's fine with me.
Once there was beautiful, shiny and very useful think-tank with an agency named NSA in a country called the US ... WTF?! ... Ok, you're right. No more fairy tales.
Actually at some point I realized how perfectly useful a filesystem like btrfs can be here. As an enterprise-level filesystem it has features like subvolumes and snapshoting among othesr but these two are useful here. Subvolumes are much like separate filesystem sharing the same storage space and snapshots are copies of other subvolumes. The great thing about those snapshots is they share storage space for unmodified data and both snapshot and original subvolume are writable.
The actual idea is to create such a directory structure:
Code: Select all
i486/
13.0/
13.1/
13.37/
x86_64/
13.0/
13.1/
13.37/
Getting to the point
Now to make that possible or a lot easier (primarily as in few work), I wrote some tools.
- chroot-helper: Prepares are chroot, actually changes root and cleans it up afterwards (based on the switch32 script I posted somewhere else)
- chroot-installer: Takes a Salix Installer ISO image and installs Salix to some arbitary directory for usage with chroots. As parts from the setup are reused, the installation (almost) completely resembles a normal installation (this was a requirement during development). Only the parts about bootloader, partitions setup, fstab, ... have been skipped as they don't make much sense when installling to a directory which might not even be a partition. So if you want your system to be actually bootable, you have to do that yourself.
- pad: This tool helps you managing your btrfs pool with the subvolumes/chroots and the snapshots/tags. Apart you can easily perform actions on sets of chroots like commands or package installations.
Here's a package which you can download that includes all 3 tools + documentation (man page):
http://gaia.homelinux.org/salix/chroot- ... ch-1ab.txz
Requires btrfs-progs
Read the man page for detailed information. If somethings missing there let me know, so I can help you and fix it for the future. I recommend also checking out the btrfs command and the filesystem in general.
Example
Apart from that here's an exemplary walkthrough:
If you have no (big enough) spare partition (otherwise use your device instead of /dev/loop2):
Code: Select all
truncate -s 30G test-btrfs.img
losetup -f test-btrfs.img
losetup -a # get the loop device for that file, here: /dev/loop2
Code: Select all
mkfs.btrfs /dev/loop2
mount /dev/loop2 /mnt
mkdir /mnt/{i486,x86_64}
btrfs subvolume create /mnt/i486/13.37
btrfs subvolume create /mnt/x86_64/13.37
Now we're going to use chroot-installer for installing Salix to those subvolumes and create a snapshot in case we mess up (don't use dashes in the names):
Code: Select all
chroot-installer --root=/mnt/i486/13.37 --iso=/home/shador/Downloads/salix-xfce-13.37.iso --kernel=smp --kb=de
btrfs subvolume snapshot /mnt/i486/13.37/ /mnt/i486/13.37_orig
chroot-installer --root=/mnt/x86_64/13.37 --iso=/home/shador/Downloads/salix64-xfce-13.37.iso --kernel= --kb=de
btrfs subvolume snapshot /mnt/x86_64/13.37/ /mnt/x86_64/13.37_orig
Now change the config of pad with your favourite editor (POOL and CHROOT_USER should be enough to start with):
Code: Select all
vi /etc/pad/default.conf
pad l
Code: Select all
i486/13.37
i486/13.37_orig
x86_64/13.37
x86_64/13.37_orig
Code: Select all
i486/13.0
i486/13.0.2a
i486/13.0.2a_orig
i486/13.1
i486/13.1.2
i486/13.1.2_orig
i486/13.37
i486/13.37_orig
x86_64/13.0
x86_64/13.0.2a
x86_64/13.0.2a_orig
x86_64/13.1
x86_64/13.1.2a
x86_64/13.1.2a_orig
x86_64/13.37
x86_64/13.37_orig
Now we're going to prepare the systems:
Code: Select all
# changes the mirror, you wouldn't want mine as it's slow for everybody else
pad e - 13.37 % sed -e 's#http://salix.enialis.net#http://gaia.homelinux.org/salix/mirror#' -i /etc/slapt-get/slapt-getrc
# upgrades the system and install some always needed stuff
pad e - 13.37 % slapt-get --add-keys \;slapt-get -u \; slapt-get -upgrade \; slapt-get -i slk-pkgcheck
# that's for my user, your uid and user name are probably different (be sure to match at least the uid)
pad e - 13.37 $ useradd -g 100 -G users,lp,floppy,audio,video,cdrom,plugdev,power,netdev,scanner -M -N -u 1992 shador
# updates root's bashrc so the command prompt indicates the chroot
pad p - 13.37
Code: Select all
if [ "$CHROOT" == "true" ]; then
if [ -z "$CHROOT_NAME" ]; then
CHROOT_NAME=chroot
fi
PS1="($CHROOT_NAME) $PS1"
fi
Done! Now you can start packaging.
To build a package, I could now do this for example:
Code: Select all
cd ~/projects/salix/build/mktoc/
screen
pad c mktoc 13.37 %% chardet,python
slkbuild -X
# oops, missed setuptools
su
slapt-get -i setuptools
exit
slkbuild -X
To clean up again issue:
Code: Select all
pad d --
Code: Select all
pad d mktoc 13.37
At the very end I would be pleased about any kind of feedback or questions especially remaining bugs and feature request (or better patches ).