Archy's Blog

I've been using DNSSEC to sign my internal DNS zones with FreeIPA for quite some time now and have never had any problems. But suddenly, I noticed ipa-dnskeysyncd.service was continuously failing to start, throwing this traceback: dnssec-keyfromlabel: fatal: failed to get key RSASHA256: not found ipa-dnskeysyncd.service: Main process exited, code=exited, status=1/FAILURE Looking deeper into the logs, I could see that LDAP was happily reporting it was adding key metadata to my zones, but the local token was instantly rejecting it: The public key was not found at: pkcs11:object=... The root cause turned out to be a cryptographic "split-brain" situation. The metadata stored in the LDAP directory was correct, but the actual cryptographic private/public key material stored by the local SoftHSM token was missing. You can verify if your saved keys are actually in the token by pointing pkcs11-tool at the FreeIPA SoftHSM database: [root@ipa03 ~]# export SOFTHSM2_CONF=/e...

For the past week, my ISP has been struggling with connectivity to resources hosted on the Cloudflare network. The issues manifested as: Packet loss High latency Low download speeds (despite upload speeds remaining unaffected) To mitigate this, I set up a split tunnel VPN to route all Cloudflare traffic through a WireGuard connection on my MikroTik router. Here is a quick guide on the setup. First, obtain a WireGuard configuration file from your VPN provider. Ensure this file contains all necessary connection parameters, including the private key, endpoint address, and public key. Below is an example of the typical configuration data you will need: [Interface] PrivateKey = <private-key-data> Address = 10.2.0.2/32 DNS = 10.2.0.1 [Peer] PublicKey = <public-key-data> AllowedIPs = 0.0.0.0/0, ::/0 Endpoint = <vpn_provider_endpoint_address>:<vpn_provider_endpoint_port> PersistentKeepalive = 25 Create the WireGuard interface on the router. ...

         NOTE: this guide exists for Upgrading from v3.15 to v3.16 as well -->  here With the current version, the  official documentation  is quite good and can be referenced. I would recommend executing all of these commands in a tmux session so that your session will remain on the server in case anything happens to your workstation. Start by checking for running tasks that would prohibit an update: [root@katello01 ~]# foreman-rake katello:upgrade_check Next, update the katello host and reboot if yum tells you to: [root@katello01 ~]# dnf -y --refresh upgrade [root@katello01 ~]# dnf needs-restarting -r If there were any updates to foreman-related packages, make sure foreman is in a consistent state: [root@katello01 ~]# foreman-maintain service stop [root@katello01 ~]# foreman-installer --scenario katello When the katello services have started again, upgrade the release-rpms: [root@katello01 ~]# dnf -y --refresh upgrade https://...

When deploying the kube-prometheus-stack on Talos Linux, you might notice that ETCD metrics are missing by default. This occurs because Talos secures ETCD using mTLS, and the default Prometheus configuration does not have the necessary certificates to authenticate against the ETCD endpoints. Here is a quick guide on how to extract the necessary certificates and configure the monitoring stack to scrape ETCD metrics successfully. First, we need to export the client certificates from a Talos control-plane node. These certificates are required for Prometheus to authenticate with ETCD. Run the following commands to copy the certificate authority, server certificate, and key to your local machine: [archy@admin42 ~]$ mkdir -p -m 700 ~/etcd [archy@admin42 ~]$ MASTER_NODE=master01.talos.archyslife.lan [archy@admin42 ~]$ talosctl -e ${MASTER_NODE} -n ${MASTER_NODE} copy /system/secrets/etcd/ca.crt ~/etcd [archy@admin42 ~]$ talosctl -e ${MASTER_NODE} -n ${MASTER_NODE} copy /system/secre...

        NOTE: this guide exists for Upgrading from v3.14 to v3.15 as well -->  here With the current version, the  official documentation  is quite good and can be referenced. I would recommend executing all of these commands in a tmux session so that your session will remain on the server in case anything happens to your workstation. Start by checking for running tasks that would prohibit an update: [root@katello01 ~]# foreman-rake katello:upgrade_check Next, update the katello host and reboot if yum tells you to: [root@katello01 ~]# dnf -y --refresh upgrade [root@katello01 ~]# dnf needs-restarting -r If there were any updates to foreman-related packages, make sure foreman is in a consistent state: [root@katello01 ~]# foreman-maintain service stop [root@katello01 ~]# foreman-installer --scenario katello When the katello services have started again, upgrade the release-rpms: [root@katello01 ~]# dnf -y --refresh upgrade https://yum.th...

For over a month now, I've been using openSUSE Tumbleweed as my daily driver, and it's been an enlightening experience. As a long-time Fedora user, I was curious about what Tumbleweed had to offer, especially since it often seems to fly under the radar in the wider Linux community. I think it’s a seriously underestimated distribution, so I wanted to share my thoughts on making the switch. The Lure of the Roll: Why I Left Fedora I've been a dedicated Fedora user since 2018, with only a brief hiatus on a MacBook. While I love Fedora's stability, the concept of a rolling release distribution that was thoroughly tested and polished always intrigued me. I pictured openSUSE Tumbleweed as something like a 'rolling release Fedora.' While Tumbleweed is famous for its seamless Btrfs integration, I stick to a traditional LVM with EXT4 partitioning scheme on my laptop. For me, the biggest draw was the option of an officially supported LTS kernel. This is a fantastic feature...

       NOTE: this guide exists for Upgrading from v3.13 to v3.14 as well -->  here With the current version, the  official documentation  is quite good and can be referenced. I would recommend executing all of these commands in a tmux session so that your session will remain on the server in case anything happens to your workstation. Start by checking for running tasks that would prohibit an update: [root@katello01 ~]# foreman-rake katello:upgrade_check Next, update the katello host and reboot if yum tells you to: [root@katello01 ~]# dnf -y --refresh upgrade [root@katello01 ~]# dnf needs-restarting -r If there were any updates to foreman-related packages, make sure foreman is in a consistent state: [root@katello01 ~]# foreman-maintain service stop [root@katello01 ~]# foreman-installer --scenario katello When the katello services have started again, upgrade the release-rpms: [root@katello01 ~]# dnf -y --refresh upgrade https://yum.the...

I'm using plain KVM + Libvirt as my hypervisor of choice in my Homelab since it gives me a lot of flexibility, reliability and performance. Installing VMs using traditional installers allows for customizations during install but if all you're doing is quickly spinning up a VM to test something, pre-built Cloud Images are probably a better choice.  The Cloud Images can be customized though before importing them using tools like virt-sysprep or cloud-init. In this article, I'll be covering a workflow using provided Cloud Images and Cloud Init to bootstrap ephemeral Linux Servers. First, we'll have to download the cloud image, I'll be using a Amazonlinux Cloud Image this time: [root@hyv02 ~]# curl -4 -f -k -L -Z -o '/var/kvm/nfs-vm-templates/amazonlinux-2023-2025-07-21-x86_64.qcow2' -X 'GET' -H 'Accept: application/octet-stream' -H 'User-Agent: curl/1.33.7' https://cdn.amazonlinux.com/al2023/os-images/2023.8.20250721.2/kvm/al2023-kvm-2...

I'm using plain KVM + Libvirt as my hypervisor of choice in my Homelab since it gives me a lot of flexibility, reliability and performance. Installing VMs using traditional installers allows for customizations during install but if all you're doing is quickly spinning up a VM to test something, pre-built Cloud Images are probably a better choice.  The Cloud Images can be customized though before importing them using tools like virt-sysprep or cloud-init . In this Article, I'll be covering my workflow using virt-sysprep with a Alma Cloud Image although any other cloud image should work. [root@hyv02 ~]# curl -4 -f -k -L -Z -o '/var/kvm/nfs-vm-templates/almalinux-9-2025-05-22-x86_64.qcow2' -X 'GET' -H 'Accept: application/octet-stream' https://raw.repo.almalinux.org/almalinux/9/cloud/x86_64/images/AlmaLinux-9-GenericCloud-9.6-20250522.x86_64.qcow2 [root@hyv02 ~]# chown root:root /var/kvm/nfs-vm-templates/almalinux-9-2025-05-22.x86_64.qcow2; chmod 60...

  There's an older version covering EL8 I already wrote up guides for EL7 and EL8, so I might as well write one for EL9 since I've been using it for almost 2.5 years now. I'll start with a minimal install of AlmaLinux 9 with the latest updates applied. First, install the required packages to make the host a hypervisor: [archy@hyv02 ~]$ sudo -y --refresh install qemu-kvm libvirt libguestfs-tools virt-install virt-top vmtouch tuned swtpm cockpit cockpit-machines [archy@hyv02 ~]$ sudo systemctl enable --now libvirtd.service tuned.service [archy@hyv02 ~]$ sudo tuned-adm profile virtual-host NOTE: Tuned is optional but might give you just a little bit more optimization for your workload. Next up, network configuration. I'll create a bond with 2 NICs which can then be used for vlans and bridges. [archy@hyv02 ~]$ sudo nmcli connection add type bond con-name bond0 ifname bond0 mode 802.3ad [archy@hyv02 ~]$ sudo nmcli connection add type ethernet con-name bond0-ens...