Cobbler (v3.3.7) Debian Deployment Guide

This guide assumes that you have a Fedora 34 server or workstation system configured to run Cobbler v3.3.7 similarly to the Cobbler v3.3.6 Beginner’s guide.

Objective
1. Caveats
Cobbler Server Prep
1. Dependencies
Debian 12 Bookworm PXE Deployment
Debian 11 Bullseye PXE Deployment
Mirroring with debmirror
Tips & Troubleshooting

Objective

Starting where the Beginner’s guide left off, further configure the Cobbler v3.3.7 server to deploy the latest release of Debian 11 Bullseye and Debian 12 Bookworm server via PXE network boot, using the same system and network environment. This guide assumes that you still have selinux and firewalld configured and enabled as described in the Beginner’s guide.

Caveats

Along the same lines as the Beginner’s guide I prefer to locally host/mirror the base OS repo to alleviate public internet availability and bandwidth dependencies during network installations.
This guide details 2 distinct approaches to deploying Debian with Cobbler
- The first method is my preferred approach since it’s more similar to the Beginner’s guide and generally has less requirements.
  - This method does require a fairly complicated fix to a bug in the debian-installer source code, caused by the unsigned repo mirror (imported from the CD/DVD/ISO installation media) failing security verification (more info in the Tips & Troubleshooting section below).
- For completeness, the Cobbler-recommended method of using debmirror (seen in the output of the cobbler check CLI command) will be detailed towards the bottom of this guide, in the Mirroring with debmirror section.
  - Debian actually recommends using something like “apt-cacher-ng” or “squid proxy”

Cobbler Server Prep

Debian maintainers advise that network booting Debian is not for novice users, and they’re not wrong, but we’re no amateurs.

That said, continue at your own risk.

Dependencies

While this section describes installing Debian from the local Cobbler server, the resulting system will use the public Debian mirrors for updates and installs going forward. Add a cobbler repo entry for those public Debian mirrors with your desired components to be referenced in the auto-install (these repos are only for metadata, they will not be mirrored or downloaded locally with the below settings)

cobbler repo add --name bookworm-latest --keep-updated false --mirror-locally false --breed apt --arch x86_64 --mirror "http://deb.debian.org/debian" --apt-components="main non-free non-free-firmware contrib" --apt-dists=bookworm
cobbler repo add --name bookworm-updates --keep-updated false --mirror-locally false --breed apt --arch x86_64 --mirror "http://deb.debian.org/debian" --apt-components="main non-free non-free-firmware contrib" --apt-dists=bookworm-updates
cobbler repo add --name bookworm-security --keep-updated false --mirror-locally false --breed apt --arch x86_64 --mirror "http://security.debian.org/debian-security" --apt-components="main non-free non-free-firmware contrib" --apt-dists=bookworm-security

cobbler repo add --name bullseye-latest --keep-updated false --mirror-locally false --breed apt --arch x86_64 --mirror "http://deb.debian.org/debian" --apt-components="main non-free non-free-firmware contrib" --apt-dists=bullseye
cobbler repo add --name bullseye-updates --keep-updated false --mirror-locally false --breed apt --arch x86_64 --mirror "http://deb.debian.org/debian" --apt-components="main non-free non-free-firmware contrib" --apt-dists=bullseye-updates
cobbler repo add --name bullseye-security --keep-updated false --mirror-locally false --breed apt --arch x86_64 --mirror "http://security.debian.org/debian-security" --apt-components="main non-free non-free-firmware contrib" --apt-dists=bullseye-security

Download the latest releases of Debian Bookworm and Bullseye

cd ~/Downloads && wget https://cdimage.debian.org/debian-cd/current/amd64/iso-dvd/debian-12.8.0-amd64-DVD-1.iso
cd ~/Downloads && wget https://cdimage.debian.org/mirror/cdimage/archive/11.11.0/amd64/iso-dvd/debian-11.11.0-amd64-DVD-1.iso

Additionally, all Debian releases at the time of writing require the “netboot” initial ramdisk (initrd) provided separately by the Debian maintainers in order to properly boot over the network. Furthermore, Debian 11 Bullseye and earlier require the “netboot firmware” image, which is an extension to the netboot initrd image containing drivers for non-free firmware, also provided separately by Debian.

wget -O ~/Downloads/Debian12.8-netboot.gz https://ftp.debian.org/debian/dists/stable/main/installer-amd64/current/images/netboot/debian-installer/amd64/initrd.gz

wget -O ~/Downloads/Debian11.11-netboot.gz  https://ftp.debian.org/debian/dists/bullseye/main/installer-amd64/current/images/netboot/debian-installer/amd64/initrd.gz
wget -O ~/Downloads/Debian11.11-firmware.gz https://cdimage.debian.org/cdimage/firmware/bullseye/current/firmware.cpio.gz

The netboot initrd.gz file above is not maintained in the same manner as the main Debian installation media; only the netboot initrd.gz image corresponding to the LATEST official release of Debian for any major version (bookworm, bullseye, etc…) will be available on the official FTP site above, and not the cdimage repo or archive where the .iso images are acquired from (the images CAN be found on the Debian repo snapshot archive, but you’ll have to correlate the date with the appropriate release).

All previous releases of the netboot firmware.cpio.gz image ARE available on the official cdimage archive at the time of writing.

Newer releases of the netboot initrd.gz and firmware images will not be compatible with previous releases of Debian, as they use different kernel and module versions.

Debian 12 Bookworm PXE Deployment

mount the Debian 12 Bookworm installation media and import the distro into cobbler:

[ -e /mnt/Debian ] || mkdir /mnt/Debian
mount -t iso9660 -o loop,ro ~/Downloads/debian-12.8.0-amd64-DVD-1.iso /mnt/Debian
cobbler import --name Debian12.8 --path /mnt/Debian

Concatenate the two initrd.gz images in the following order, then configure the new Cobbler Distro to use the combined image:

cat /var/www/cobbler/distro_mirror/Debian12.8/install.amd/initrd.gz ~/Downloads/Debian12.8-netboot.gz > /var/www/cobbler/pub/Debian12.8-netboot.gz

cobbler distro edit --name Debian12.8-x86_64 --initrd "/var/www/cobbler/pub/Debian12.8-netboot.gz"

Create a new “preseed” (autoinstall) template from the sample included with Cobbler, and make the necessary changes to install from an unofficial source.

cp /var/lib/cobbler/templates/sample.seed /var/lib/cobbler/templates/bookworm-workaround.seed
sed -i 's,# d-i debian-installer/allow_unauthenticated boolean true,d-i debian-installer/allow_unauthenticated boolean true,' /var/lib/cobbler/templates/bookworm-workaround.seed

Use the standard preseed_early_default Cobbler Script to create a new Cobbler Script which will implement the workaround to be run on the PXE client to circumvent the bug mentioned in the Caveats section above:

grep -v '# End preseed_early_default' /var/lib/cobbler/scripts/preseed_early_default | tee /var/lib/cobbler/scripts/preseed_early_workaround

echo 'while true; do 
	[ -f /usr/lib/base-installer/library.sh ] && sed -i '\''s/APT::Get::AllowUnauthenticated "true";/APT::Get::AllowUnauthenticated "true";\nAcquire::AllowInsecureRepositories "true";/'\'' /usr/lib/base-installer/library.sh && break;
	sleep 1;
done 2> /dev/null 1> /dev/null &' | tee -a /var/lib/cobbler/scripts/preseed_early_workaround

Create a new Cobbler snippet from late_apt_repo_config to not use the Cobbler server as an apt mirror post-installation.

grep -v 'deb http://$http_server/cblr/links/$distro_name $os_version main' /var/lib/cobbler/snippets/late_apt_repo_config | tee /var/lib/cobbler/snippets/late_apt_repo_config-no_cobbler

Create another preseed script from the standard preseed_late_default which will use the above late_apt_repo_config-no_cobbler snippet.

sed 's,late_apt_repo_config,late_apt_repo_config-no_cobbler' /var/lib/cobbler/scripts/preseed_late_default | tee /var/lib/cobbler/scripts/preseed_late_no-cobbler

Modify the new preseed (autoinstall) template to deselect any services (which are not included with the installation ISO/DVD), install desired packages only, setup an additional user “debian” with sudo privileges, set the workaround script created above to run in the early installation stage and the new “preseed_late” script to run in the late stage, and more aggressively force the installation regardless of any existing installations.

sed -i 's,# d-i apt-setup/services-select multiselect security,d-i apt-setup/services-select multiselect,' /var/lib/cobbler/templates/bookworm-workaround.seed
sed -i 's,ntp ssh wget,wget curl openssh-server sudo vim\npopularity-contest popularity-contest/participate boolean false,' /var/lib/cobbler/templates/bookworm-workaround.seed
sed -i 's,d-i partman-auto/method string lvm,d-i partman-efi/non_efi_system boolean true\nd-i partman-auto/method string lvm,' /var/lib/cobbler/templates/bookworm-workaround.seed
sed -i 's,d-i grub-installer/bootdev string default,d-i grub-installer/bootdev string default\nd-i grub-installer/with_other_os boolean true\nd-i grub-installer/force-efi-extra-removable boolean false\nd-i grub-installer/enable_os_prober_otheros_yes boolean false\nd-i grub-installer/enable_os_prober_otheros_no boolean true,' /var/lib/cobbler/templates/bookworm-workaround.seed
sed -i 's,d-i passwd/make-user boolean false,d-i passwd/make-user boolean true\nd-i passwd/user-fullname string Debian Admin\nd-i passwd/username string debian\nd-i passwd/user-password-crypted password $default_password_crypted\nd-i passwd/user-default-groups string sudo,' /var/lib/cobbler/templates/bookworm-workaround.seed
sed -i 's,script=preseed_early_default,script=preseed_early_workaround,' /var/lib/cobbler/templates/bookworm-workaround.seed
sed -i 's,script=preseed_late_default,script=preseed_late_no-cobbler,' /var/lib/cobbler/templates/bookworm-workaround.seed

Configure the Cobbler Profile to use the new autoinstall (preseed) template and assign the appropriate cobbler repo mirrors defined above:

cobbler profile edit --name Debian12.8-x86_64 --autoinstall 'bookworm-workaround.seed' --repos "bookworm-latest bookworm-security bookworm-updates"

Finally, create a new Cobbler System to PXE boot and automatically install Debian 12 Bookworm, replacing the “aa:bb:cc:dd:ee:ff” with the MAC address of your PXE client, being sure not to use a duplicate MAC or IP address of any other Cobbler System, then sync up Cobbler.

cobbler system add --name "Debian12.8" --profile Debian12.8-x86_64 --hostname "debian12-8" --mac-address "aa:bb:cc:dd:ee:ff" --netboot-enabled true --ip-address "10.0.0.12" --netmask "255.255.255.0" --gateway "10.0.0.1" --name-servers "10.0.0.1 1.1.1.1 8.8.8.8"
cobbler sync

The PXE Client VM can now be powered on, and should automatically boot to PXE and install Debian 12.8 to the VM HDD using the “bookworm-workaround.seed” preseed template created above.

Debian 11 Bullseye PXE Deployment

Take similar steps as above, with the inclusion of the netboot “firmware” image file in the combined initrd.gz image file, and using the “bookworm-workaround.seed” template created above as a starting point and trimming off a few unsupported preseed settings to import and automatically deploy Debian 11 Bullseye over PXE.

[ -e /mnt/Debian ] || mkdir /mnt/Debian
mount -t iso9660 -o loop,ro ~/Downloads/debian-11.11.0-amd64-DVD-1.iso /mnt/Debian
cobbler import --name=Debian11.11 --path=/mnt/Debian

cat /var/www/cobbler/distro_mirror/Debian11.11/install.amd/initrd.gz ~/Downloads/Debian11.11-netboot.gz ~/Downloads/Debian11.11-firmware.gz > /var/www/cobbler/pub/Debian11.11-netboot.gz
cobbler distro edit --name Debian11.11-x86_64 --initrd "/var/www/cobbler/pub/Debian11.11-netboot.gz"

grep -v "os_prober" /var/lib/cobbler/templates/bookworm-workaround.seed > /var/lib/cobbler/templates/bullseye-workaround.seed
cobbler profile edit --name Debian11.11-x86_64 --autoinstall bullseye-workaround.seed --repos "bullseye-latest bullseye-security bullseye-updates"

cobbler system add --name Debian11.11 --profile Debian11.11-x86_64 --mac-address "aa:bb:cc:dd:ee:ff" --netboot-enabled true

cobbler sync

Mirroring with debmirror

Mirroring the official Debian repos with the debmirror tool is fairly simple, but will require about 100GB of extra disk space on the Cobbler server.

This guide will only provide the steps to mirror the LATEST release of Debian (Bookworm 12.8 at the time of writing) with debmirror.

similarly to the DVD/ISO method above, both the netboot and DVD/ISO initrd.gz images are still required in the same concatenated manner to properly install over the network, but they must both be downloaded separately of the debmirror process.

Since the official Debian repo will be mirrored, the source will be fully valid with trusted signatures, meaning the autoinstall (preseed) file for the PXE client will not need the workaround portion.

Cobbler supports managing repos/sources via the debmirror command natively, which means the debmirror package only needs to be installed and configured, then mirroring can be accomplished/managed through Cobbler CLI commands.

The syntax for mirroring a Debian repo through the debmirror command will still be provided below for context in the Manual debmirror section, but is not necessary in most cases to mirror Debian locally.

Setup debmirror

Follow the instructions given from the cobbler check command for supporting Debian deployments/repos and install the Debian keyrings:

yum install -y debmirror debian-keyring
sed -i "s/@arches/#@arches/g" /etc/debmirror.conf
sed -i "s/@dists/#@dists/g" /etc/debmirror.conf
cobbler sync
cobbler check

Manual debmirror

Sometimes debmirror may fail to download a few files from the configured source, which can cause Cobbler to error out on the reposync task/command.

Usually re-issueing the reposync command (and in-turn, the debmirror command) after such a failure will complete the download of the missing files.

This manual debmirror method may be useful for troubleshooting

debmirror -p -v --method http -a amd64 -h ftp.us.debian.org -d stable -s main,main/debian-installer,contrib,non-free,non-free-firmware --nosource --keyring="/usr/share/keyrings/debian-archive-keyring.gpg"  ~/Debian12.8

Cobbler reposync and debmirror

use the cobbler repo and cobbler reposync commands in order to mirror Debian latest repo locally:

cobbler repo add --name Debian-latest --keep-updated true --mirror-locally true --breed apt --arch x86_64 --mirror "http://ftp.us.debian.org/debian" --apt-components=main,main/debian-installer,non-free-firmware --apt-dists=bookworm --yumopts '--keyring'='/usr/share/keyrings/debian-archive-keyring.gpg'
cobbler reposync

PXE deploy via local debmirror repo

Configure a new preseed file for installation via the trusted, signed Debian repo:

cp /var/lib/cobbler/templates/bookworm-workaround.seed /var/lib/cobbler/templates/bookworm-latest.seed
sed -i 's,d-i debian-installer/allow_unauthenticated,#d-i debian-installer/allow_unauthenticated,' /var/lib/cobbler/templates/bookworm-latest.seed
sed -i 's,script=preseed_early_workaround,script=preseed_early_default,' /var/lib/cobbler/templates/bookworm-latest.seed

Download the kernel and 2 initrd files for Debiain latest/current (12.8)

wget -O ~/Downloads/debian-stable-initrd.gz http://ftp.us.debian.org/debian/dists/stable/main/installer-amd64/current/images/cdrom/initrd.gz
wget -O ~/Downloads/debian-stable-netboot-initrd.gz http://ftp.us.debian.org/debian/dists/stable/main/installer-amd64/current/images/netboot/debian-installer/amd64/initrd.gz
wget -O ~/Downloads/debian-stable-vmlinuz http://ftp.us.debian.org/debian/dists/stable/main/installer-amd64/current/images/cdrom/vmlinuz
mkdir /var/www/cobbler/pub/Debian12.8-netboot
cat ~/Downloads/debian-stable-initrd.gz ~/Downloads/debian-stable-netboot-initrd.gz > /var/www/cobbler/pub/Debian12.8-netboot/initrd.gz
cp ~/Downloads/debian-stable-vmlinuz /var/www/cobbler/pub/Debian12.8-netboot/vmlinuz

Add a Cobbler Distro and Profile (including the metadata-only cobbler repo mirrors created in the Prerequisites section above) for the new Debian repo, and the usual HTTP link:

cobbler distro add --name Debian-latest --arch x86_64 --autoinstall-meta 'tree'='http://@@http_server@@/cblr/links/Debian-latest' --breed debian --os-version bookworm --initrd "/var/www/cobbler/pub/Debian12.8-netboot/initrd.gz" --kernel "/var/www/cobbler/pub/Debian12.8-netboot/vmlinuz"
cobbler profile add --name Debian-latest --distro Debian-latest --autoinstall bookworm-latest.seed --repos "bookworm-latest bookworm-security bookworm-updates"
ln -s /var/www/cobbler/repo_mirror/Debian-latest /var/www/cobbler/links/Debian-latest
cobbler sync

Now the PXE Client should be able to boot from a generic/random MAC address and load the new “Debian-latest” Cobbler Profile over GRUB

Cobbler Systems can now also be added similarly to above in the Debian 12 Bookworm PXE Deployment and Debian 11 Bullseye PXE Deployment sections above.
cobbler system add --name Debian-Latest --profile Debian-latest --mac-address "aa:bb:cc:dd:ee:ff" --netboot-enabled true

Tips & Troubleshooting

The Cobbler 3.3.6 Beginner’s Guide Tips & Troubleshooting section contains some basic recommendations and limitations of Cobbler which will not be repeated here.
Going the route of importing the DVD ISO file for debian PXE deployments will leave those PXE clients with less available packages to install during OS installation compared to a full mirror, but I prefer to keep the OS deployments minimal anyway.
The issue mentioned in the Caveats section above is due to the base-installer source code neglecting to add the Acquire::AllowInsecureRepositories "true"; line in to the 00AllowUnauthenticated file on the target system (created depending on the value of preseed optoin debian-installer/allow_unauthenticated) as mentioned in the apt-secure manpage.
- The installation will fail on a “cannot find a suitable kernel package” error.
- This error consistenly occurs on Debian 11 & 12 installations.
- A bug has been filed with the Debian Bug Tracking System for this.
- Debian developers do NOT sign the software/repos packaged into official CD/DVD/ISO releases: https://wiki.debian.org/SecureApt
References