Cobbler (v3.3.7) XCP-ng Deployment Guide

This guide assumes that you have a Fedora 34 server or workstation system configured to run Cobbler v3.3.7 similarly to the Cobbler v3.3.7 Beginner’s guide.

Objective
1. Caveats
Cobbler Server Prep
1. Dependencies
XCP-ng 8.2.1 PXE Deployment
1. Standard Kernel
2. Alternate Kernel
  1. PXE Boot Alt Kernel
  2. Install Alt Kernel
Tips & Troubleshooting

Objective

Starting where the Beginner’s guide left off, further configure the Cobbler v3.3.7 server to deploy XCP-ng v8.2.1 via PXE network boot, using the same system and network environment with the exception of reconfiguring the PXE Client VM to have 64 GB HDD in order to meet the due to the minimum requirements of XCP-ng 8.2.1. This guide assumes that you still have selinux and firewalld configured and enabled as described in the Beginner’s guide.

Caveats

Cobbler’s support for XCP-ng not up-to-date, necessitating several custom scripts and templates for proper supported_arches
XCP-ng 8.2.1 uses a custom fork of a fairly old linux kernel, so this guide will provide several solutions to customizing the installation process and resulting systems for supporting different hardware.

Cobbler Server Prep

Cobbler v3.3.7 is not natively designed to support XCP-ng v8.2.1 and will require some code changes in order to import the distro properly. Some Cobbler sync-triggers will also be required to correct the kernel command line parameters in the bootloader configs.

Dependencies

Download the official installation media for XCP-ng 8.2.1:

cd ~/Downloads && wget https://updates.xcp-ng.org/isos/8.2/xcp-ng-8.2.1-20231130.iso

Define the OS Version by inserting the following code-block in /var/lib/cobbler/distro_signatures.json immediately following the entry named xcp16 which is included with Cobbler

      "xcp821": {
        "signatures": [
          "Packages"
        ],
        "version_file": "^xcp-ng-release-8\\.2\\.1.*\\.x86_64\\.rpm$",
        "version_file_regex": null,
        "kernel_arch": "xen\\.gz",
        "kernel_arch_regex": "^.*(x86_64).*$",
        "supported_arches": [
          "x86_64"
        ],
        "supported_repo_breeds": [],
        "kernel_file": "vmlinuz",
        "initrd_file": "xen\\.gz",
        "isolinux_ok": false,
        "default_autoinstall": "answerfile.xml",
        "kernel_options": "dom0_max_vcpus=1-2 dom0_mem=max:752M,752M",
        "kernel_options_post": "",
        "boot_files": [
          "install.img"
        ]
      },

With this modification made, be careful not to use the cobbler signature update command on the Cobbler server, which will remove any user-defined signatures.

Edit the import_signatures.py script included with Cobbler in order to import the new Distro

START=$(cat -n /usr/local/lib/python3.9/site-packages/cobbler/modules/managers/import_signatures.py | grep "krn_lines = self.get_file_lines(os.path.join(dirname, x))" | awk {' print $1 '})
head -n $(echo $(($START-1))) /usr/local/lib/python3.9/site-packages/cobbler/modules/managers/import_signatures.py > /tmp/import_signatures.py
END=$(cat -n /usr/local/lib/python3.9/site-packages/cobbler/modules/managers/import_signatures.py | grep "if m:" | awk {' print $1 '})
cat << EOF >> /tmp/import_signatures.py
                    ftype_krn = magic.detect_from_filename(os.path.join(dirname, x))
                    if ftype_krn.mime_type == "application/gzip":
                        with gzip.open(os.path.join(dirname, x), 'r') as f:
                            krn_lines = f.readlines()
                    else:
                        krn_lines = self.get_file_lines(os.path.join(dirname, x))
                    for line in krn_lines:
                        try:
                            m = re_krn2.match(line)
                        except TypeError:
                            #  regex evaluatoin "string pattern" failed... is it bytes?
                            # https://docs.python.org/3.12/library/codecs.html
                            try:
                                m = re_krn2.match(line.decode("UTF-8"))
                            except UnicodeDecodeError:
                                try:
                                    m = re_krn2.match(line.decode("ASCII"))
                                except UnicodeDecodeError:
                                    try:
                                        m = re_krn2.match(line.decode("latin-1"))
                                    except:
                                        pass
                                except:
                                    pass
                            except:
                                pass
                        except:
                            pass
EOF
tail -n+$END /usr/local/lib/python3.9/site-packages/cobbler/modules/managers/import_signatures.py >> /tmp/import_signatures.py 
cp /tmp/import_signatures.py /usr/local/lib/python3.9/site-packages/cobbler/modules/managers/.
rm -f /tmp/import_signatures.py && START="" && END=""

Create the [answerfile.xml] file with the following contents and save it to /var/lib/cobbler/templates/answerfile.xml

<?xml version="1.0"?>
 <installation sr-type="lvm" mode="fresh" netinstall-gpg-check="true">
  <keymap>us</keymap>
  <timezone>US/Eastern</timezone>
  <source type="url">$tree/</source>
  <driver-source type="url">$tree/</driver-source>
  <primary-disk>sda</primary-disk>
  <root-password type="hash">$default_password_crypted</root-password>
  <bootloader location="mbr">grub2</bootloader>
$SNIPPET('network_config_XCP')  <ntp-server>us.pool.ntp.org</ntp-server>
  <network-backend>vswitch</network-backend>
  <post-install-script type="url">
## Figure out if we're automating OS installation for a system or a profile
#if $getVar('system_name','') != ''
#set $what = "system"
#else
#set $what = "profile"
#end if
   http://$http_server/cblr/svc/op/script/$what/$name/?script=XCP-post-install.sh
  </post-install-script>
 </installation>

Create the Cobbler Snippet called by the autoinstall template above:

/var/lib/cobbler/snippets/network_config_XCP

## start of cobbler network_config generated code
#if $getVar("system_name","") != ""
    #set ikeys = $interfaces.keys()
    #import re
    #for $iname in $ikeys
        #set $idata = $interfaces[$iname]
        ## Ignore BMC interface
        #if $idata["interface_type"].lower() == "bmc"
            #continue
        #end if
    #end for
    #for $iname in $ikeys
        #set $idata    = $interfaces[$iname]
        #set $mac      = $idata["mac_address"]
        #set $static   = $idata["static"]
        #set $ip       = $idata["ip_address"]
        #set $netmask  = $idata["netmask"]
        #set $type     = $idata["interface_type"]
        ## Ignore BMC interface
        #if $type == "bmc"
            #continue
        #end if
        #if $mac != ""
            #if $static == True:
                #if $ip != "":
  <admin-interface name="$iname" proto="static">
   <ipaddr>$ip</ipaddr>
                    #if $netmask != "":
   <subnet>$netmask</subnet>
                    #end if
                #end if
                #if $gateway != "":
   <gateway>$gateway</gateway>
  </admin-interface>
                #else
  </admin-interface>
                #end if
                #if $name_servers and $name_servers[0] != "":
                    ## Anaconda only allows one nameserver
  <name-server>$name_servers[0]</name-server>
                #end if
            #else
  <admin-interface name="$iname" proto="dhcp" />
            #end if
        #else
  <admin-interface name="$iname" proto="dhcp" />
        #end if
        #if $hostname != ""
  <hostname>$hostname</hostname>
        #else
            #set $myhostname = $getVar('name','').replace("_","-")
  <hostname>$myhostname</hostname>
        #end if
    #end for
#else
## profile based install so just provide one interface for starters
#set $myhostname = $getVar('hostname',$getVar('name','cobbler')).replace("_","-")
  <admin-interface name="eth0" proto="dhcp" />
  <hostname>$myhostname</hostname>
#end if

Create the post-install-script referenced in the autoinstall (answerfile.xml) template created above:

/var/lib/cobbler/scripts/XCP-post-install.sh

#!/bin/bash
yum -c $1/etc/yum.conf --installroot $1 update -y
yum -c $1/etc/yum.conf --installroot $1 install -y vim ipref
chroot $1 touch /root/test.txt
# # # *** OR *** # # #
# echo "touch /root/test.txt" | chroot $1 /bin/bash -s

Create the following Cobbler sync-triggers which will correct the bootloader configurations for both GRUB and PXELINUX for all XCP-ng 8.2.1 Profiles and Systems (saved to the filesystem location noted above the code-block):

/var/lib/cobbler/triggers/sync/post/fix-XCP-BIOS_CSM-PXE.sh

#!/bin/bash
for PROFILE in $(cobbler profile list); do
    DIST=$(cobbler profile report --name $PROFILE | grep Distribution | awk {' print $3 '});
    VER=$(cobbler distro report --name $DIST | grep "OS Version" | awk {' print $4 '});
    [[ ($VER == 'xcp821' || $VER == 'xcp83') ]] || continue;
    LINE=$(cat -n /var/lib/tftpboot/pxelinux.cfg/default | grep -m 1 "MENU LABEL $PROFILE" | awk {' print $1'});
    head -n $LINE /var/lib/tftpboot/pxelinux.cfg/default > /tmp/default;
    LINE=$(echo $(($(tail -n+$(echo $(($LINE+1))) /var/lib/tftpboot/pxelinux.cfg/default | grep -n -m 1 ^[LABEL\|MENU] | awk {' print $1 '} | sed 's,:.*,,')+$LINE)));
    echo "        kernel mboot.c32" >> /tmp/default;
    echo "        append /images/$DIST/xen.gz dom0_max_vcpus=2 dom0_mem=2048M,max:2048M com1=115200,8n1 console=com1,vga --- /images/$DIST/vmlinuz xencons=hvc console=hvc0 console=tty0 --- /images/$DIST/install.img" >> /tmp/default
    echo "        ipappend 2" >> /tmp/default;
    tail -n+$LINE /var/lib/tftpboot/pxelinux.cfg/default >> /tmp/default;
    LINE="" && DIST="" && VER="";
    cp /tmp/default /var/lib/tftpboot/pxelinux.cfg/. && rm -f /tmp/default;
done

/var/lib/cobbler/triggers/sync/post/fix-XCP-GRUB.sh

#!/bin/bash
for PROFILE in $(cobbler profile list); do 
    DIST=$(cobbler profile report --name $PROFILE | grep Distribution | awk {' print $3 '});
    VER=$(cobbler distro report --name $DIST | grep "OS Version" | awk {' print $4 '});
    [[ ($VER == 'xcp821' || $VER == 'xcp83') ]] || continue;
    LINE=$(cat -n /var/lib/tftpboot/grub/x86_64_menu_items.cfg | grep "menuentry '$PROFILE" | awk {' print $1'});
    head -n $LINE /var/lib/tftpboot/grub/x86_64_menu_items.cfg > /tmp/x86_64_menu_items.cfg;
    LINE=$(echo $(($(tail -n+$(echo $(($LINE+1))) /var/lib/tftpboot/grub/x86_64_menu_items.cfg | grep -n -m 1 ^} | awk {' print $1 '} | sed 's,:.*,,')+$LINE)));
    echo "  echo 'chainloading...?'" >> /tmp/x86_64_menu_items.cfg;
    echo "  multiboot2 /images/$DIST/xen.gz dom0_mem=2048M,max:2048M watchdog dom0_max_vcpus=4 com1=115200,8n1 console=com1,vga" >> /tmp/x86_64_menu_items.cfg;
    echo "  echo 'loading kernel...'" >> /tmp/x86_64_menu_items.cfg;
    echo "  module2 /images/$DIST/vmlinuz console=hvc0 console=tty0 answerfile=http://10.0.0.10/cblr/svc/op/autoinstall/profile/$PROFILE install" >> /tmp/x86_64_menu_items.cfg;
    echo "  echo 'loading initial ramdisk...'" >> /tmp/x86_64_menu_items.cfg;
    echo "  module2 /images/$DIST/install.img" >> /tmp/x86_64_menu_items.cfg;
    echo "  echo 'done'" >> /tmp/x86_64_menu_items.cfg;
    tail -n+$LINE /var/lib/tftpboot/grub/x86_64_menu_items.cfg >> /tmp/x86_64_menu_items.cfg;
    LINE="" && DIST="" && VER="";
    cp /tmp/x86_64_menu_items.cfg /var/lib/tftpboot/grub/. && rm -f /tmp/x86_64_menu_items.cfg;
done

/var/lib/cobbler/triggers/sync/post/fix-XCP-systems-GRUB.sh

#!/bin/bash
for SYSTEM in $(cobbler system list); do
    PROFILE=$(cobbler system report --name $SYSTEM | grep ^Profile | awk {' print $3 '});
    DIST=$(cobbler profile report --name $PROFILE | grep ^Distribution | awk {' print $3 '});
    BREED=$(cobbler distro report --name $DIST | grep ^Breed | awk {' print $3 '});
    if [[ $BREED == "xen" ]]; then
        MAC="$(cobbler system report --name $SYSTEM | grep ^MAC\ Address | awk {' print $4 '})"
        [ $MAC ] || continue
        LINE=$(cat -n /var/lib/tftpboot/grub/system/$MAC | grep menuentry\ \'$SYSTEM | awk {' print $1 '})
        head -n $LINE /var/lib/tftpboot/grub/system/$MAC > /tmp/"$MAC"
        LINE=$(echo $(($(tail -n+$(echo $(($LINE+1))) /var/lib/tftpboot/grub/system/$MAC | grep -n -m 1 ^} | awk {' print $1 '} | sed 's,:.*,,')+$LINE)))
        echo "  echo 'chainloading...?'" >> /tmp/"$MAC"
        echo "  multiboot2 /images/$DIST/xen.gz dom0_mem=2048M,max:2048M watchdog dom0_max_vcpus=4 com1=115200,8n1 console=com1,vga" >> /tmp/"$MAC"
        echo "  echo 'loading kernel...'" >> /tmp/"$MAC"
        echo "  module2 /images/$DIST/vmlinuz console=hvc0 console=tty0 answerfile=http://10.0.0.10/cblr/svc/op/autoinstall/system/$SYSTEM install" >> /tmp/"$MAC"
        echo "  echo 'loading initial ramdisk...'" >> /tmp/"$MAC"
        echo "  module2 /images/$DIST/install.img" >> /tmp/"$MAC"
        echo "  echo 'done'" >> /tmp/"$MAC"
        tail -n+$LINE /var/lib/tftpboot/grub/system/$MAC >> /tmp/"$MAC"
        cp /tmp/"$MAC" /var/lib/tftpboot/grub/system/"$MAC" && rm -f /tmp/"$MAC"
        LINE="" && MAC="" && BREED="" && PROFILE="" && DIST=""
    fi;
done

/var/lib/cobbler/triggers/sync/post/fix-XCP-systems-PXE.sh

#!/bin/bash
for SYSTEM in $(cobbler system list); do
    PROFILE=$(cobbler system report --name $SYSTEM | grep ^Profile | awk {' print $3 '});
    DIST=$(cobbler profile report --name $PROFILE | grep ^Distribution | awk {' print $3 '});
    BREED=$(cobbler distro report --name $DIST | grep ^Breed | awk {' print $3 '});
    if [[ $BREED == "xen" ]]; then
        MAC="$(cobbler system report --name $SYSTEM | grep ^MAC\ Address | awk {' print $4 '} | sed 's,:,-,g')"
        LINE=$(cat -n /var/lib/tftpboot/pxelinux.cfg/01-$MAC | grep -m 1 "MENU LABEL $SYSTEM" | awk {' print $1'})
        head -n $LINE /var/lib/tftpboot/pxelinux.cfg/01-$MAC > /tmp/01-$MAC
        echo "        kernel mboot.c32" >> /tmp/01-$MAC
        echo "        append /images/$DIST/xen.gz dom0_max_vcpus=2 dom0_mem=2048M,max:2048M com1=115200,8n1 console=com1,vga --- /images/$DIST/vmlinuz xencons=hvc console=hvc0 console=tty0 answerfile=http://10.0.0.10/cblr/svc/op/autoinstall/system/$SYSTEM install --- /images/$DIST/install.img" >> /tmp/01-$MAC
        echo "        ipappend 2" >> /tmp/01-$MAC
        cp /tmp/01-$MAC /var/lib/tftpboot/pxelinux.cfg/. && rm -f /tmp/01-$MAC
        LINE="" && MAC="" && PROFILE="" && DIST="" && BREED=""
    fi;
done

Mark the 4 Cobbler sync-triggers as executable:

chmod u+x /var/lib/cobbler/triggers/sync/post/fix-XCP-BIOS_CSM-PXE.sh
chmod u+x /var/lib/cobbler/triggers/sync/post/fix-XCP-GRUB.sh
chmod u+x /var/lib/cobbler/triggers/sync/post/fix-XCP-systems-PXE.sh
chmod u+x /var/lib/cobbler/triggers/sync/post/fix-XCP-systems-GRUB.sh

Sync up Cobbler to apply the changes

systemctl restart cobblerd && sleep 10
cobbler sync

XCP-ng 8.2.1 PXE Deployment

XCP-ng 8.2.1 comes with 2 linux kernels available to install, this guide will detail both how to boot and load either kernel over the network from the PXE Client AND install either kernel to the PXE Client and configure it as the defaul on the resulting XCP-ng 8.2.1 installation.

Standard Kernel

Mount the installation media and run cobbler import

mkdir /mnt/XCP-NG
mount -t iso9660 -o loop,ro ~/Downloads/xcp-ng-8.2.1-20231130.iso /mnt/XCP-NG
cobbler import --name XCP --path /mnt/XCP-NG

Link some additional bootloader resources to the cobbler loaders folder, and sync up cobbler:

XCP-ng official docs recommends using the mboot.c32 & pxelinux.0 files from the XCP-ng installation media, but that shouldn’t be necessary.

ln -s /usr/share/syslinux/mboot.c32 /var/lib/cobbler/loaders/mboot.c32
systemctl restart cobblerd && sleep 10
cobbler mkloaders && sleep 5
cobbler sync

At this point, you should be able to boot the PXE Client VM and select the “XCP-x86_64” Profile from the PXE or GRUB menu to test a generic installation

Create a new Cobbler System to automatically boot and install XCP-ng 8.2.1, replacing aa:bb:cc:dd:ee:ff with the MAC Address of the PXE Client VM, and sync up Cobbler

cobbler system add --name XCP-ng --profile XCP-x86_64 --netboot-enabled true --hostname xcp-ng --interface eth0 --static true --mac-address "aa:bb:cc:dd:ee:ff" --ip-address 10.0.0.23 --gateway 10.0.0.1 --netmask 255.255.255.0 --name-servers "10.0.0.1 1.1.1.1 10.0.0.10"
systemctl restart cobblerd && sleep 10
cobbler sync

Now, boot the PXE Client and it should automatically load the XCP-ng 8.2.1 installer and complete the installation without intervention. If there is an error, try either booting or installing the alternate kernel, or more than likely, both in the section below.

Alternate Kernel

Optionally, follow the below 2 sections to either boot the alternate kernel via PXE, install the alternate kernel to the target system, or both

PXE Boot Alt Kernel

Clone the Standard Kernel Distro and reconfigure it to PXE boot the alternate kernel:

cobbler distro copy --name XCP-x86_64 --newname XCP-alt-x86_64
cobbler distro edit --name XCP-alt-x86_64 --kernel '/var/www/cobbler/distro_mirror/XCP/boot/alt/vmlinuz'

# # # This step isn't really necessary, just think its weird that Cobbler creates the new "links" web endpoint, but doesn't edit the "tree" autoinstall metadata variable (nor does it create a new copy of the mirror in distro_mirror folder)
sudo cobbler distro edit --name XCP-alt-x86_64 --autoinstall-meta 'tree'='http://@@http_server@@/cblr/links/XCP-alt-x86_64'

Add a new Cobbler Profile which uses the newly cloned Distro:

cobbler profile add --name XCP-alt-x86_64 --distro XCP-alt-x86_64

create a new System which will boot the alternate kernel, replacing *“aa:bb:cc:dd:ee:ff” with the MAC Address of the PXE Client VM, and sync up Cobbler:

cobbler system add --name XCP-alt --profile XCP-alt-x86_64 --netboot-enabled true --mac-address "aa:bb:cc:dd:ee:ff"
systemctl restart cobblerd && sleep 10
cobbler sync

Install Alt Kernel

Create a script in the following location with the below contents:

/var/lib/cobbler/scripts/XCP-alt-post-install.sh

#!/bin/bash
## echo "touch /root/test.txt" | chroot $1 /bin/bash -s
yum -c /root/yum.conf --installroot $1 install -y kernel-alt | tee /tmp/yum.txt
GRUB=\$(grep "Adding 'XCP-ng kernel-alt 4.19.265' as grub entry #" /tmp/yum.txt | awk -F "#" {' print $2 '})
sed -i "s/default=0/default=\${GRUB}/" $1/boot/efi/EFI/xenserver/grub.cfg
yum -c $1/etc/yum.conf --installroot $1 update -y
yum -c $1/etc/yum.conf --installroot $1 install -y vim ipref

Create a new autoinstall (answerfile.xml) template file from the one created above and configure the cloned alt Profile to use it, then sync up Cobbler:

sed 's,/?script=XCP-post-install.sh,/?script=XCP-alt-post-install.sh,' /var/lib/cobbler/templates/answerfile.xml | tee /var/lib/cobbler/templates/XCP-alt-answerfile.xml
cobbler profile edit --name XCP-alt-x86_64 --autoinstall XCP-alt-answerfile.xml

systemctl restart cobblerd && sleep 10
cobbler sync

Now the “XCP-alt” System created above will boot the alternate kernel over PXE and install the alt kernel to the PXE client as the default boot option.

Alternatively, to boot the installer with the standard kernel and install the alt kernel to the PXE client, create a new (or edit an existing…) Cobbler System (or Distro+Profile) to use the XCP-alt-post-install.sh script above in the autoinstall template file:
cobbler system edit --name XCP-alt --profile XCP-x86_64 --autoinstall XCP-alt-answerfile.xml

Tips & Troubleshooting

The Cobbler 3.3.7 Beginner’s Guide Tips & Troubleshooting section contains some basic recommendations and limitations of Cobbler which will not be repeated here.
References: