Archy's Blog

I'm using plain KVM + Libvirt as my hypervisor of choice in my Homelab since it gives me a lot of flexibility, reliability and performance. Installing VMs using traditional installers allows for customizations during install but if all you're doing is quickly spinning up a VM to test something, pre-built Cloud Images are probably a better choice.  The Cloud Images can be customized though before importing them using tools like virt-sysprep or cloud-init. In this article, I'll be covering a workflow using provided Cloud Images and Cloud Init to bootstrap ephemeral Linux Servers. First, we'll have to download the cloud image, I'll be using a Amazonlinux Cloud Image this time: [root@hyv02 ~]# curl -4 -f -k -L -Z -o '/var/kvm/nfs-vm-templates/amazonlinux-2023-2025-07-21-x86_64.qcow2' -X 'GET' -H 'Accept: application/octet-stream' -H 'User-Agent: curl/1.33.7' https://cdn.amazonlinux.com/al2023/os-images/2023.8.20250721.2/kvm/al2023-kvm-2...

I'm using plain KVM + Libvirt as my hypervisor of choice in my Homelab since it gives me a lot of flexibility, reliability and performance. Installing VMs using traditional installers allows for customizations during install but if all you're doing is quickly spinning up a VM to test something, pre-built Cloud Images are probably a better choice.  The Cloud Images can be customized though before importing them using tools like virt-sysprep or cloud-init . In this Article, I'll be covering my workflow using virt-sysprep with a Alma Cloud Image although any other cloud image should work. [root@hyv02 ~]# curl -4 -f -k -L -Z -o '/var/kvm/nfs-vm-templates/almalinux-9-2025-05-22-x86_64.qcow2' -X 'GET' -H 'Accept: application/octet-stream' https://raw.repo.almalinux.org/almalinux/9/cloud/x86_64/images/AlmaLinux-9-GenericCloud-9.6-20250522.x86_64.qcow2 [root@hyv02 ~]# chown root:root /var/kvm/nfs-vm-templates/almalinux-9-2025-05-22.x86_64.qcow2; chmod 60...

  There's an older version covering EL8 I already wrote up guides for EL7 and EL8, so I might as well write one for EL9 since I've been using it for almost 2.5 years now. I'll start with a minimal install of AlmaLinux 9 with the latest updates applied. First, install the required packages to make the host a hypervisor: [archy@hyv02 ~]$ sudo -y --refresh install qemu-kvm libvirt libguestfs-tools virt-install tuned swtpm cockpit cockpit-machines [archy@hyv02 ~]$ sudo systemctl enable --now libvirtd.service tuned.service [archy@hyv02 ~]$ sudo tuned-adm profile virtual-host NOTE: Tuned is optional but might give you just a little bit more optimization for your workload. Next up, network configuration. I'll create a bond with 2 NICs which can then be used for vlans and bridges. [archy@hyv02 ~]$ sudo nmcli connection add type bond con-name bond0 ifname bond0 mode 802.3ad [archy@hyv02 ~]$ sudo nmcli connection add type ethernet con-name bond0-ens2f0 ifname ens5f0...

If a system does not boot anymore, it's usually easiest to boot from a live-media and chroot into the installation to troubleshoot the issue at hand. I'll be using the archlinux installation iso to chroot into a debian install to fix a kernel update that's gone sideways. Once in the arch installation, make sure that the disks are all detected: root@archiso ~ # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop0 7:0 0 853.9M 1 loop /run/archiso/airootfs sr0 11:0 1 1.2G 0 rom /run/archiso/bootmnt vda 254:0 0 32G 0 disk ├─vda1 254:1 0 976M 0 part └─vda2 254:2 0 31G 0 part ├─vg_base-lv_root 253:0 0 3.8G 0 lvm ├─vg_base-lv_usr 253:1 0 5.7G 0 lvm ├─vg_base-lv_var 253:2 0 3.8G 0 lvm ├─vg_base-lv_var_log 253:3 0 3.8G 0 lvm ├─vg_base-lv_var_tmp 253:4 0 1.9G 0 lvm ├─vg_base-lv_tmp 253:5 0 488M 0 lvm ├─vg_base-lv_home 253:6 0 976M 0 lvm └─vg_base-lv_...

I've recently run into a issue where the GPU Operator prevented the Machine Config Operator to apply Cluster Updates because of not being able to unload the Driver. In my case, the nodename was 'cl1gpu08.cluster.example.com' since it's going to be referenced in some commands. The fix was actually simple. First, disable the GPU Operator on the node: $ oc label node/cl1gpu08.cluster.prod.example.com nvidia.com/gpu.deploy.operands=false Next, make sure there are no NVIDIA GPU Operator Workloads running on that gpu: $ oc -n nvidia-gpu-operator get pods -o wide --field-selector spec.nodeName=cl1gpu08.cluster.prod.example.com If you're impatient, you can go ahead and remove the remaining pods as well as restart the machine-config-daemon. Once the node is back, set the label to 'true' so that the GPU Operator can be scheduled again on that node: $ oc label node/cl1gpu08.cluster.prod.example.com nvidia.com/gpu.deploy.operands- --overwrite Sources used:   - ...

After upgrading to Foreman 3.14 / Katello 4.16  I've had issues syncing ansible collections and jobs failed with the following error: Error message: the server returns an error HTTP status code: 502 Response headers: {"date"=>"Sun, 20 Apr 2025 14:30:42 GMT", "server"=>"Apache", "content-length"=>"341", "content-type"=>"text/html; charset=iso-8859-1"} Response body: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>502 Proxy Error</title> </head><body> <h1>Proxy Error</h1> <p>The proxy server received an invalid response from an upstream server.<br /> The proxy server could not handle the request<p>Reason: <strong>Error reading from remote server</strong></p></p> </body></html> Also, there's a Foreman Discourse Threa...

      NOTE: this guide exists for Upgrading from v3.12 to v3.13 as well -->  here With the current version, the  official documentation  is quite good and can be referenced. I would recommend executing all of these commands in a tmux session so that your session will remain on the server in case anything happens to your workstation. Start by checking for running tasks that would prohibit an update: [root@katello01 ~]# foreman-rake katello:upgrade_check Next, update the katello host and reboot if yum tells you to: [root@katello01 ~]# dnf -4 --refresh upgrade [root@katello01 ~]# dnf needs-restarting -r If there were any updates to foreman-related packages, make sure foreman is in a consistent state: [root@katello01 ~]# foreman-maintain service stop [root@katello01 ~]# foreman-installer --scenario katello When the katello services have started again, upgrade the release-rpms: [root@katello01 ~]# dnf -y --refresh upgrade https://yum.theforema...

While upgrading my Foreman+Katello Server to the latest version , I've encountered the following issue: [archy@katello01 ~]$ sudo foreman-installer --scenario katello 2025-04-20 11:25:08 [NOTICE] [root] Loading installer configuration. This will take some time. 2025-04-20 11:25:12 [NOTICE] [root] Running installer with log based terminal output at level NOTICE. 2025-04-20 11:25:12 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions. 2025-04-20 11:25:14 [NOTICE] [checks] System checks passed 2025-04-20 11:25:21 [NOTICE] [pre] The Foreman database foreman does not exist. 2025-04-20 11:25:21 [NOTICE] [configure] Starting system configuration. 2025-04-20 11:25:31 [NOTICE] [configure] 250 configuration steps out of 1939 steps complete. 2025-04-20 11:25:34 [NOTICE] [configure] 500 configuration steps out of 1940 steps complete. 2025-04-20 11:25:42 [NOTICE] [configure] 1000 configu...

The prerequisites include configuring a local registry mirror for OpenShift images and ensuring a functional OpenShift cluster with administrative privileges.  Optionally , if you are using a custom certificate authority (CA), a ConfigMap containing the CA certificate must be configured in the 'openshift-config' namespace under the 'updateservice-registry' key. For example: [archy@helper01 ~]$ oc -n openshift-config get configmap/custom-ca -o yaml apiVersion: v1 kind: ConfigMap metadata: namespace: openshift-config name: custom-ca spec: ca-bundle.crt: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- updateservice-registry: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- You can add additional trusted CAs by updating the 'image.config.openshift.io/cluster' resource: [archy@helper01 ~]$ oc patch image.config.openshift.io/cluster -p '{"spec":{"additionalTruste...