Cobbler (v3.3.7) Debian Deployment Guide

This guide assumes that you have a Fedora 34 server or workstation system configured to run Cobbler v3.3.7 similarly to the Cobbler v3.3.7 Beginner’s guide.

Objective
1. Caveats
Cobbler Server Prep
1. Dependencies
Debian 12 Bookworm PXE Deployment
Debian 11 Bullseye PXE Deployment
Mirroring with debmirror
Tips & Troubleshooting

Objective

Starting where the Beginner’s guide left off, further configure the Cobbler v3.3.7 server to deploy the latest release of Debian 11 Bullseye and Debian 12 Bookworm via PXE network boot, using the same system and network environment. This guide assumes that you still have selinux and firewalld configured and enabled as described in the Beginner’s guide.

Caveats

using the cobbler check CLI command, the output recommends installing the debmirror package in order to manage Debian deployments and repos with Cobbler. Instead, we will simply import the Debian source installation media (.iso file), which will become available over HTTP, and use that as the local repo mirror to install over the network.
- This method is not the Cobbler official method, and Debian actually recommends using something like “apt-cacher-ng” or “squid proxy”
- The Main Debian repo for the latest release of Debian is about 100 GB at the time of writing, which is why this is my preferred method over using debmirror, as it only requires the data from a single Debian installer DVD (.iso file)
  - For completeness however, the debmirror method will be detailed towards the bottom of this guide, in the Mirroring with debmirror section.
This guide provides a workaround to an issue that is encountered when installing Debian 11 & 12 over the network from a local/private repo where the PXE client (debian-installer) does not trust the installation source (Since Debian developers DO NOT sign the software/repos packaged into official CD/DVD/ISO releases: https://wiki.debian.org/SecureApt ).
- This appears to be a bug in the debian-installer (d-i); more info provided in the Tips & Troubleshooting section below.
- This issue should not arise when mirroring an official Debian mirror using the debmirror method, as recommended by Cobbler

Cobbler Server Prep

Debian maintainers advise that network booting Debian is not for novice users, and they’re not wrong, but we’re no amateurs.

That said, continue at your own risk.

Dependencies

Download the latest releases of Debian Bookworm and Bullseye

cd ~/Downloads && wget https://cdimage.debian.org/debian-cd/current/amd64/iso-dvd/debian-12.8.0-amd64-DVD-1.iso
cd ~/Downloads && wget https://cdimage.debian.org/mirror/cdimage/archive/11.11.0/amd64/iso-dvd/debian-11.11.0-amd64-DVD-1.iso

Additionally, all Debian releases at the time of writing require the “netboot” initial ramdisk (initrd) provided separately by the Debian maintainers in order to properly boot over the network. Furthermore, Debian 11 Bullseye and earlier require the “netboot firmware” image, which is an extension to the netboot initrd image containing drivers for non-free firmware, also provided by Debian.

wget -O ~/Downloads/Debian12.8-netboot.gz https://ftp.debian.org/debian/dists/stable/main/installer-amd64/current/images/netboot/debian-installer/amd64/initrd.gz

wget -O ~/Downloads/Debian11.11-netboot.gz  https://ftp.debian.org/debian/dists/bullseye/main/installer-amd64/current/images/netboot/debian-installer/amd64/initrd.gz
wget -O ~/Downloads/Debian11.11-firmware.gz https://cdimage.debian.org/cdimage/firmware/bullseye/current/firmware.cpio.gz

The netboot initrd.gz file above is not maintained in the same manner as the main Debian installation media; only the netboot initrd.gz image corresponding to the LATEST official release of Debian for any major version (bookworm, bullseye, etc…) will be available on the official FTP site above, and not the cdimage repo or archive where the .iso images are acquired from (the images CAN be found on the Debian repo snapshot archive, but you’ll have to correlate the date with the appropriate release).

All previous releases of the netboot firmware.cpio.gz image ARE available on the official cdimage archive at the time of writing.

Newer releases of the netboot initrd.gz and firmware images will not be compatible with previous releases of Debian, as they use different kernel and module versions.

Create a GPG key to sign the local Debian repo mirror, and then export the public key to the public HTTP share on the Cobbler server and restore the SELinux labels.

gpg --full-gen-key --batch <(echo "Key-Type: 1"; \
                             echo "Key-Length: 4096"; \
                             echo "Subkey-Type: 1"; \
                             echo "Subkey-Length: 4096"; \
                             echo "Expire-Date: 0"; \
                             echo "Name-Real: cobbler"; \
                             echo "Name-Email: cobbler@fedora.local"; \
                             echo "%no-protection"; )

gpg --output /var/www/cobbler/pub/cobbler.gpg.asc --armor --export cobbler
sudo restorecon -R /var/www/cobbler/pub

for simplicity, this GPG key will not be password protected. This is against cyber-security best practices and should only be implemented in a test environment.

Debian 12 Bookworm PXE Deployment

mount the Debian 12 Bookworm installation media and import the distro into cobbler:

[ -e /mnt/Debian ] || mkdir /mnt/Debian
mount -t iso9660 -o loop,ro /home/fedora/Downloads/debian-12.8.0-amd64-DVD-1.iso /mnt/Debian
cobbler import --name Debian12.8 --path /mnt/Debian

Navigate to the imported distro mirror and sign the repo:

cd /var/www/cobbler/distro_mirror/Debian12.8/dists/bookworm
gpg -u cobbler -bao Release.gpg Release
gpg -u cobbler --clear-sign --output InRelease Release

Concatenate the two initrd.gz images in the following order, then configure the new Cobbler Distro to use the combined image:

cat /var/www/cobbler/distro_mirror/Debian12.8/install.amd/initrd.gz ~/Downloads/Debian12.8-netboot.gz > /var/www/cobbler/pub/Debian12.8-netboot.gz

cobbler distro edit --name Debian12.8-x86_64 --initrd "/var/www/cobbler/pub/Debian12.8-netboot.gz"

Create a new “preseed” (autoinstall) template from the sample included with Cobbler, and make the necessary changes to install from an unofficial source.

cp /var/lib/cobbler/templates/sample.seed /var/lib/cobbler/templates/bookworm-sample.seed
sed -i 's,# d-i debian-installer/allow_unauthenticated boolean true,d-i debian-installer/allow_unauthenticated boolean true,' /var/lib/cobbler/templates/bookworm-sample.seed
sed -i 's,# d-i apt-setup/local0/key string http://local.server/key,d-i apt-setup/local0/repository string http://$http_server$install_source_directory\nd-i apt-setup/local0/key string http://$http_server/cblr/pub/cobbler.gpg.asc,' /var/lib/cobbler/templates/bookworm-sample.seed

This is where the bug mentioned in the Caveats section above is introduced. The apt-setup/local0/key preseed value should allow us to provide the debian-installer with a GPG key to use to validate the installation source.

Still confugre it to retrieve the key over HTTP anyway, just in case.

Create a file named preseed_early_workaround in the /var/lib/cobbler/scripts directory with the following contents which will run on the PXE client to circumvent the bug mentioned in the Caveats section above:

#!/bin/sh
while true; do
	[ -d /target/etc/apt/trusted.gpg.d ] && cd /target/etc/apt/trusted.gpg.d && fetch-url http://10.0.0.10/cblr/pub/cobbler.gpg.asc cobbler.gpg.asc && exit;
	sleep 1;
done 2> /dev/null 1> /dev/null &

Optionally, create another preseed script (/var/lib/cobbler/scripts/preseed_late_sudo_sources) to run in the post-installation stage which will grant sudo privileges to the “debian” user (created below) and reconfigure the installed system to use Debian’s official APT sources instead of our local Cobbler server.

#!/bin/sh
sed -i 's,#PermitRootLogin prohibit-password,PermitRootLogin no,' /etc/ssh/sshd_config
usermod -aG sudo debian
sed -i 's,^,#,' /etc/apt/sources.list
echo 'deb http://deb.debian.org/debian bookworm main non-free-firmware non-free contrib' >> /etc/apt/sources.list
echo 'deb-src http://deb.debian.org/debian bookworm main non-free-firmware non-free contrib' >> /etc/apt/sources.list
echo 'deb http://security.debian.org/debian-security bookworm-security main non-free-firmware non-free contrib' >> /etc/apt/sources.list
echo 'deb-src http://security.debian.org/debian-security bookworm-security main non-free-firmware non-free contrib' >> /etc/apt/sources.list
echo 'deb http://deb.debian.org/debian bookworm-updates main non-free-firmware non-free contrib' >> /etc/apt/sources.list
echo 'deb-src http://deb.debian.org/debian bookworm-updates main non-free-firmware non-free contrib' >> /etc/apt/sources.list

Modify the new preseed (autoinstall) template to install desired packages only, setup an additional user “debian”, set the workaround script created above to run in the “early” installation stage, and the “preseed_late” script to run in the late stage, if configured, and more aggressively force the installation regardless of any existing installations.

sed -i 's,ntp ssh wget,wget curl openssh-server sudo vim\npopularity-contest popularity-contest/participate boolean false,' /var/lib/cobbler/templates/bookworm-sample.seed
sed -i 's,# d-i apt-setup/services-select multiselect security,d-i apt-setup/services-select multiselect,' /var/lib/cobbler/templates/bookworm-sample.seed
sed -i 's,d-i partman-auto/method string lvm,d-i partman-efi/non_efi_system boolean true\nd-i partman-auto/method string lvm,' /var/lib/cobbler/templates/bookworm-sample.seed
sed -i 's,d-i grub-installer/bootdev string default,d-i grub-installer/bootdev string default\nd-i grub-installer/with_other_os boolean true\nd-i grub-installer/force-efi-extra-removable boolean false\nd-i grub-installer/enable_os_prober_otheros_yes boolean false\nd-i grub-installer/enable_os_prober_otheros_no boolean true,' /var/lib/cobbler/templates/bookworm-sample.seed
sed -i 's,d-i passwd/make-user boolean false,d-i passwd/make-user boolean true\nd-i passwd/user-fullname string Debian Admin\nd-i passwd/username string debian\nd-i passwd/user-password-crypted password $default_password_crypted,' /var/lib/cobbler/templates/bookworm-sample.seed
sed -i 's,script=preseed_early_default,script=preseed_early_workaround,' /var/lib/cobbler/templates/bookworm-sample.seed
sed -i 's,script=preseed_late_default,script=preseed_late_sudo_sources,' /var/lib/cobbler/templates/bookworm-sample.seed

Configure the Cobbler Profile to use the new autoinstall (preseed) template:

cobbler profile edit --name Debian12.8-x86_64 --autoinstall 'bookworm-sample.seed'

Finally, create a new Cobbler System to PXE boot and automatically install Debian 12 Bookworm, replacing the “aa:bb:cc:dd:ee:ff” with the MAC address of your PXE client, being sure not to use a duplicate MAC or IP addresse of any other Cobbler System, then sync up Cobbler.

cobbler system add --name "Debian12.8" --profile Debian12.8-x86_64 --hostname "debian12-8" --mac-address "aa:bb:cc:dd:ee:ff" --netboot-enabled true --ip-address "10.0.0.12" --netmask "255.255.255.0" --gateway "10.0.0.1" --name-servers "10.0.0.1 1.1.1.1 8.8.8.8"
cobbler sync

The PXE Client VM can now be powered on, and should automatically boot to PXE and install Debian 12.8 to the VM HDD using the “bookworm-sample.seed” preseed template created above.

Debian 11 Bullseye PXE Deployment

Take similar steps as above, with the inclusion of the netboot “firmware” image file in the combined initrd.gz image file, and using the “bookworm-sample.seed” template created above as a starting point and trimming off a few unsupported preseed settings to import and automatically deploy Debian 11 Bullseye over PXE.

[ -e /mnt/Debian ] || mkdir /mnt/Debian
mount -t iso9660 -o loop,ro /home/fedora/Downloads/debian-11.11.0-amd64-DVD-1.iso /mnt/Debian
cobbler import --name=Debian11.11 --path=/mnt/Debian

cd /var/www/cobbler/distro_mirror/Debian11.11/dists/bullseye
gpg -u cobbler -bao Release.gpg Release
gpg -u cobbler --clear-sign --output InRelease Release

cat /var/www/cobbler/distro_mirror/Debian11.11/install.amd/initrd.gz ~/Downloads/Debian11.11-netboot.gz ~/Downloads/Debian11.11-firmware.gz > /var/www/cobbler/pub/Debian11.11-netboot.gz
cobbler distro edit --name Debian11.11-x86_64 --initrd "/var/www/cobbler/pub/Debian11.11-netboot.gz"

grep -v "os_prober" /var/lib/cobbler/templates/bookworm-sample.seed > /var/lib/cobbler/templates/bullseye-sample.seed
cobbler profile edit --name Debian11.11-x86_64 --autoinstall bullseye-sample.seed

cobbler system add --name Debian11.11 --profile Debian11.11-x86_64 --mac-address "aa:bb:cc:dd:ee:ff" --netboot-enabled true

cobbler sync

Mirroring with debmirror

Mirroring the official Debian repos with the debmirror tool is fairly simple, but will require about 100GB of extra disk space on the Cobbler server.

This guide will only provide the steps to mirror the LATEST release of Debian (Bookworm 12.8 at the time of writing) with debmirror.

similarly to the DVD/ISO method above, both the netboot and DVD/ISO initrd.gz images are still required in the same concatenated manner to properly install over the network, but they must both be downloaded separately of the debmirror process.

Since the official Debian repo will be mirrored, the source will be fully valid with trusted signatures, meaning the autoinstall (preseed) file for the PXE client will not need the APT security settings required in the DVD/ISO source method.

Cobbler supports managing repos/sources via the debmirror command natively, which means the debmirror package only needs to be installed and configured, then mirroring can be accomplished/managed through Cobbler CLI commands.

The syntax for mirroring a Debian repo through the debmirror command will still be provided below for context in the Manual debmirror section, but is not necessary in most cases to mirror Debian locally.

Setup debmirror

Follow the instructions for supporting Debian deployments/repos given from the cobbler check command output (as well as installign Debian keyrings):

yum install -y debmirror debian-keyring
sed -i "s/@arches/#@arches/g" /etc/debmirror.conf
sed -i "s/@dists/#@dists/g" /etc/debmirror.conf
cobbler sync
cobbler check

Manual debmirror

Sometimes debmirror may fail to download a few files from the configured source, which can cause Cobbler to error out on the reposync task/command.

Usually re-issueing the reposync command (and in-turn, the debmirror command) after such a failure will complete the download of the missing files.

This manual debmirror method may be useful for troubleshooting

debmirror -p -v --method http -a amd64 -h ftp.us.debian.org -d stable -s main,main/debian-installer,contrib,non-free,non-free-firmware --nosource --keyring="/usr/share/keyrings/debian-archive-keyring.gpg"  ~/Debian12.8

Cobbler reposync and debmirror

use the cobbler repo and cobbler reposync commands in order to mirror Debian lates repo locally:

cobbler repo add --name Debian-latest --keep-updated true --mirror-locally true --breed apt --arch x86_64 --mirror "http://ftp.us.debian.org/debian" --apt-components=main,main/debian-installer,non-free-firmware --apt-dists=bookworm --yumopts '--keyring'='/usr/share/keyrings/debian-archive-keyring.gpg'
cobbler reposync

Configure a new preseed file for installation via the trusted, signed Debian repo:

cp /var/lib/cobbler/templates/bookworm-sample.seed /var/lib/cobbler/templates/bookworm-latest.seed
sed -i 's,d-i apt-setup/local0/repository,#d-i apt-setup/local0/repository,' /var/lib/cobbler/templates/bookworm-latest.seed
sed -i 's,d-i apt-setup/local0/key,#d-i apt-setup/local0/key,' /var/lib/cobbler/templates/bookworm-latest.seed
sed -i 's,d-i debian-installer/allow_unauthenticated,#d-i debian-installer/allow_unauthenticated,' /var/lib/cobbler/templates/bookworm-latest.seed
sed -i 's,script=preseed_early_workaround,script=preseed_early_skip,' /var/lib/cobbler/templates/bookworm-latest.seed
cp /var/lib/cobbler/scripts/preseed_early_workaround /var/lib/cobbler/scripts/preseed_early_skip
echo -e '#!/bin/sh\nexit' | tee /var/lib/cobbler/scripts/preseed_early_skip

Download the kernel and 2 initrd files for Debiain latest/current (12.8)

wget -O ~/Downloads/debian-stable-initrd.gz http://ftp.us.debian.org/debian/dists/stable/main/installer-amd64/current/images/cdrom/initrd.gz
wget -O ~/Downloads/debian-stable-netboot-initrd.gz http://ftp.us.debian.org/debian/dists/stable/main/installer-amd64/current/images/netboot/debian-installer/amd64/initrd.gz
wget -O ~/Downloads/debian-stable-vmlinuz http://ftp.us.debian.org/debian/dists/stable/main/installer-amd64/current/images/cdrom/vmlinuz
mkdir /var/www/cobbler/pub/Debian12.8-netboot
cat /home/fedora/Downloads/debian-stable-initrd.gz /home/fedora/Downloads/debian-stable-netboot-initrd.gz > /var/www/cobbler/pub/Debian12.8-netboot/initrd.gz
cp ~/Downloads/debian-stable-vmlinuz /var/www/cobbler/pub/Debian12.8-netboot/vmlinuz

Add a Cobbler Distro and Profile for the new Debian repo, and the usual HTTP link:

cobbler distro add --name Debian-latest --arch x86_64 --autoinstall-meta 'tree'='http://@@http_server@@/cblr/links/Debian-latest' --breed debian --os-version bookworm --initrd "/var/www/cobbler/pub/Debian12.8-netboot/initrd.gz" --kernel "/var/www/cobbler/pub/Debian12.8-netboot/vmlinuz"
cobbler profile add --name Debian-latest --distro Debian-latest --autoinstall bookworm-latest.seed
ln -s /var/www/cobbler/repo_mirror/Debian-latest /var/www/cobbler/links/Debian-latest
cobbler sync

Now the PXE Client should be able to boot from a generic/random MAC address and load the new “Debian-latest” Cobbler Profile over GRUB

Cobbler Systems can now also be added similarly to above in the Debian 12 Bookworm PXE Deployment and Debian 11 Bullseye PXE Deployment sections above.
cobbler system add --name Debian-Latest --profile Debian-latest --mac-address "aa:bb:cc:dd:ee:ff" --netboot-enabled true

Tips & Troubleshooting

The Cobbler 3.3.7 Beginner’s Guide Tips & Troubleshooting section contains some basic recommendations and limitations of Cobbler which will not be repeated here.
The debian-installer issue mentioned in the Caveats section above is primarily marked by a fatal error during the installation process and the log entry GPG error in /var/log/syslog on the installer system, or, if the workaround script was used, you can install the debconf-utils package on the resulting system and use the command debconf-get-selections --installer | grep "Downloading" for more info on the error.
- The error prevents the installation from proceeding using the configured repo mirror, but the debian-installer remains running; press ALT+F2 or ALT+F3 on the PXE client keyboard to access a privileged shell to conduct some troubleshooting.
- The issue stems from the below section of the preseed file, retrieved drectly from Debian’s official stable and bookworm example preseed files.

### Apt setup
# Choose, if you want to scan additional installation media
# (default: false).
d-i apt-setup/cdrom/set-first boolean false
# You can choose to install non-free firmware.
#d-i apt-setup/non-free-firmware boolean true
# You can choose to install non-free and contrib software.
#d-i apt-setup/non-free boolean true
#d-i apt-setup/contrib boolean true
# Uncomment the following line, if you don't want to have the sources.list
# entry for a DVD/BD installation image active in the installed system
# (entries for netinst or CD images will be disabled anyway, regardless of
# this setting).
#d-i apt-setup/disable-cdrom-entries boolean true
# Uncomment this if you don't want to use a network mirror.
#d-i apt-setup/use_mirror boolean false
# Select which update services to use; define the mirrors to be used.
# Values shown below are the normal defaults.
#d-i apt-setup/services-select multiselect security, updates
#d-i apt-setup/security_host string security.debian.org

# Additional repositories, local[0-9] available
#d-i apt-setup/local0/repository string \
#       http://local.server/debian stable main
#d-i apt-setup/local0/comment string local server
# Enable deb-src lines
#d-i apt-setup/local0/source boolean true
# URL to the public key of the local repository; you must provide a key or
# apt will complain about the unauthenticated repository and so the
# sources.list line will be left commented out.
#d-i apt-setup/local0/key string http://local.server/key
# or one can provide it in-line by base64 encoding the contents of the
# key file (with `base64 -w0`) and specifying it thus:
#d-i apt-setup/local0/key string base64://LS0tLS1CRUdJTiBQR1AgUFVCTElDIEtFWSBCTE9DSy0tLS0tCi4uLgo=
# The content of the key file is checked to see if it appears to be ASCII-armoured.
# If so it will be saved with an ".asc" extension, otherwise it gets a '.gpg' extension.
# "keybox database" format is currently not supported. (see generators/60local in apt-setup's source)

# By default the installer requires that repositories be authenticated
# using a known gpg key. This setting can be used to disable that
# authentication. Warning: Insecure, not recommended.
#d-i debian-installer/allow_unauthenticated boolean true

(continued)
- As seen in the procedures above, allow_unauthenticated is set to true, which appears to only permit the installation to START when an untrusted repo mirror is found, but the mirror will be checked again before beginning the apt-setup phase, which allow_unauthenticated will not override.
  - This means that the debian-installer also requires the local APT mirror to be defined using the d-i apt-setup/local0/repository & d-i apt-setup/local0/key preseed options in order for the PXE client to retrieve the public GPG key for the repo mirror and use it to validate the mirror, which seems to be configured properly through the preseed from Cobbler, verified with the following commands on the debian-installer system (and obviously can be done, since the workaround provided in this guide does essentially the same thing):
    debconf-get apt-setup/local0/repository debconf-get apt-setup/local0/key less /var/lib/preseed/log cat /var/log/syslog | grep "GPG error" cat /var/lib/cdebconf/templates.dat | grep apt-setup cat /var/lib/cdebconf/questions.dat | grep apt-setup
  - Alternatively, the preseed comments note that you may provide the GPG key inline in the preseed file as a base64-encoded string with the format d-i apt-setup/local0/key string base64://<base64-encoded_key>, however I was not able to get this working either.
- The snippet from Debian’s official preseed sample above mentions “see generators/60local in apt-setup’s source” and looking at some of that source code, I tried to model the workaround script as closely to the source code as possible, which is why fetch-url is used to download the GPG key, instead of wget or curl.
- This error consistenly occurs on Debian 11 & 12 installations.
References