Archy's Blog

Since Foreman 3.3 and Katello 4.5 are the last supported versions on EL7, I decided to do an in-place upgrade from CentOS 7 to Almalinux 8 using leapp. I'm using a fully up-to-date (as of 19th September 2022) installation of Foreman 3.3, Katello 4.5. So first, make sure you're up to date with your installation: [root@katello01 ~]# yum -y clean all [root@katello01 ~]# rm -rf /var/cache/yum [root@katello01 ~]# yum -y makecache fast [root@katello01 ~]# yum -y update If any foreman-packages have been updated, go ahead and ensure that your foreman-installation is consistent: [root@katello01 ~]# foreman-maintain service stop [root@katello01 ~]# foreman-installer --scenario katello Make sure that the installer runs without errors before continuing. Check if you need to reboot the server and reboot it if package updates require it [root@katello01 ~]# yum needs-restarting -r [root@katello01 ~]# reboot The foreman team is hosting a patched version of the leapp upg...

Using a default storage class will allow you to deploy volumes as they are claimed / requested without any manual intervention. Kubernetes provides multiple built-in storage provisioners but most of them are focused on cloud environments and since I run my cluster on-prem with nfs as shared storage, I will have to use the nfs-subdir external provisioner ( github ).  This will allow me to share one directory and the provisioner will create a directory on the nfs share for each volume that is being allocated. For more documentation, check the link above. First, clone the git repository: [ 17:25:09 ] - archy ~> git clone https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner.git git/kubernetes/prod/nfs-subdir-external-provisioner [ 17:25:10 ] - archy ~> cd git/kubernetes/prod/nfs-subdir-external-provisioner Create the RBAC Bindings required for the provisioner to work: [ 17:25:24 ] - archy ~/git/kubernetes/prod/nfs-subdir-external-provisioner - master> kube...

Rsync is a very easy way to transfer large amounts of files between servers with low resource usage which also makes for a great backup target due to its speed and flexibility. Here's a small write-up on how to set up a basic rsyncd server: First, install the required packages: [archy@mirror01~]$ sudo yum -y --refresh install rsync rsync-daemon Create the rsync root directory, I'll place all my sub-modules (folders) in '/var/rsync' [archy@mirror01 ~]$ mkdir -p /var/rsync/{rpms,backup} Setting the SELinux boolean and file context: [archy@mirror01 ~]$ sudo semanage boolean -m -1 rsync_full_access [archy@mirror01 ~]$ sudo semanage fcontext -a -t 'public_content_t' '/var/rsync(/.*)?' [archy@mirror01 ~]$ sudo restorecon -Rv /var/rsync Creating /etc/rsyncd.conf: [archy@mirror01 ~]$ sudo vim /etc/rsyncd.conf pid file = /var/run/rsyncd.pid log file = /var/log/rsyncd.log lock file = /var/run/rsyncd.lock address = 0.0.0.0 port = 873 ...

I encountered the issue that when provisioning hosts, they are missing most of their repositories. Checking the logs and Content Host itself didn't help any further since everything seemed to be in order. For reference, I'm running these packages: [root@katello01 ~]# rpm -qa | grep -iE '^pulp|^candlepin|^foreman|^katello' | grep -viE 'archyslife' foreman-cli-3.3.0-1.el7.noarch katello-selinux-4.0.2-1.el7.noarch katello-debug-4.5.0-1.el7.noarch pulp-client-1.0-1.noarch pulpcore-selinux-1.3.2-1.el7.x86_64 foreman-postgresql-3.3.0-1.el7.noarch foreman-installer-katello-3.3.0-1.el7.noarch foreman-dynflow-sidekiq-3.3.0-1.el7.noarch katello-client-bootstrap-1.7.9-1.el7.noarch katello-server-ca-1.0-1.noarch foreman-release-3.3.0-1.el7.noarch candlepin-4.1.11-1.el7.noarch candlepin-selinux-4.1.11-1.el7.noarch foreman-installer-3.3.0-1.el7.noarch foreman-3.3.0-1.el7.noarch katello-certs-tools-2.9.0-1.el7.noarch katello-...

A broken rpmdb can happen when packages are updated outside of rpm / yum being notified and therefore the rpmdb being in an inconsistent state or if the partition where your rpmdb resides runs out of disk space. This is the error I'm getting: error: rpmdb: BDB0113 Thread/process 6245/139652028413760 failed: BDB1507 Thread died in Berkeley DB library error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery error: cannot open Packages index using db5 - (-30973) error: cannot open Packages database in /var/lib/rpm CRITICAL:yum.main: Error: rpmdb open failed Fixing it is very simple. First, create a backup of the current rpmdb just in case things go sideways: [root@server ~]# tar -cvpPJf /var/tmp/rpmdb-$(date +%F).tar.xz /var/lib/rpm Now that you have a backup, remove the rpmdb files: [root@server ~]# rm -f /var/lib/rpm/__db.* With the rpmdb deleted, go ahead and rebuild rpm's db: [root@server ~]# rpm -qa -...

Systemd-Timers are a more flexible alternative to cronjobs. For example, Systemd will take care of the dependencies and the logging of your timer whereas with cron you'd have to program it yourself. In this example, I will use systemd-timers to create a backup of the zabbix PostgreSQL database. First, create the service that will actually run the task: [root@zabbix ~]# cat << EOF >> /etc/systemd/system/zabbix-db-backup.service [Unit] Description=Create a Backup of the zabbix database After=network-online.target postgresql.service [Service] Type=simple Nice=19 ExecStart=/usr/bin/sh /root/scripts/zabbix-db-backup.sh User=root Group=root EOF Now create the timer unit which is responsible for scheduling your service: [root@zabbix ~]# cat << EOF >> /etc/systemd/system/zabbix-db-backup.timer [Unit] Description=Timer for zabbix-db-backup.service [Timer] OnCalendar=*-*-* *:45:00 Unit=zabbix-db-backup.service Persiste...

When debugging network-related Issues, it might be helpful to set the MTU to a static value instead of leaving it on 'auto'. On Distros that utilize NetworkManager, this can easily be done using 'nmcli'. Here's a simple configuration for a previously unconfigured interface called 'eno1': [root@server ~]# nmcli connection modify eno1 802-3-ethernet.mtu 1518 [root@server ~]# nmcli connection modify eno1 ipv4.address 192.0.2.5/24 [root@server ~]# nmcli connection modify eno1 ipv4.method manual [root@server ~]# nmcli connection modify eno1 ipv6.method ignore [root@server ~]# nmcli connection up eno1 This will set the MTU, a static IP address, and ignore any ipv6 addressing. Now that the interface is up, check the MTU using the 'ip' command: [root@server ~]# ip -s -c -h link show eno1 | egrep -i 'mtu' 6: eno1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1518 qdisc mq state UP mode DEFAULT group default qlen 1000 As you can s...

NOTE: This guide exists for Upgrading from v3.1 to v3.2 as well --> here When updating your foreman installation manually, I find that the official documentation  is a bit lacking. So here's how I update my foreman-server from v3.2 to v3.3 and katello from v4.4 to v4.5. I would recommend executing all of these commands in a tmux session so that your session will remain on the server in case anything happens to your workstation. Start by checking for running tasks that would prohibit an update: [root@katello01 ~]# foreman-rake katello:upgrade_check Next, update the katello host and reboot it if yum tells you to: [root@katello01 ~]# yum -d 2 -y update [root@katello01 ~]# needs-restarting -r When the katello services have started again, upgrade the repository: [root@katello01 ~]# yum -d 2 -y install https://yum.theforeman.org/releases/3.3/el7/x86_64/foreman-release.rpm [root@katello01 ~]# yum -d 2 -y install https://yum.theforeman.org/katello/4.5/katello/el7/x86_64/kat...

When updating Rundeck from version 4.0 to 4.1(+), the database has to be migrated from v1 to v2 because the service won't start otherwise. So here's  a quick how-to: First, stop the rundeck service and create a backup: [root@rundeck ~]# systemctl stop rundeckd.service [root@rundeck ~]# mkdir -p /var/backup/rundeck [root@rundeck ~]# tar -cvpf /var/backup/rundeck/rundeck-db-v1-$(date +%F).tar /var/lib/rundeck/data There's a script that can be utilized to migrate the db from v1 to v2 so let's clone that: [root@rundeck ~]# git clone https://github.com/rundeck-plugins/h2-v2-migration.git Now run the migration script against the current database. Ensure that you have a backup so you can rollback if things go haywire: [root@rundeck ~]# cd h2-v2-migration [root@rundeck ~]# /usr/bin/sh migration.sh -f /var/lib/rundeck/data/rundeckdb -u 'sa' -p Once that migration is done, the script will create an './output' directory where the db files are stor...

This error most likely occurs during startups and is usually caused by errors or missing parameters in the service unit configuration. In my case, the service provided by a vendor, let's call it 'fancy.service', was misconfigured out of the box which led to these errors in the system's journal: [root@server ~]# journalctl ... Jun 21 20:02:54 server.example.com systemd[992]: fancy.service: Failed to determine user credentials: No such process Jun 21 20:02:54 server.example.com systemd[992]: fancy.service: Failed at step USER spawning /usr/bin/sh: No such process ... The root cause was the order in which the services were started. This 'fancy.service' didn't have any 'Wants=' or 'After=' declared although it required having network (network.target) access and being able to authenticate users (sssd.service), so let's fix that. First, create the overrides-directory for the service and add the required settings to a '.conf...

Depending on what you're configuring, you might have to configure YAML Variables inside of text strings that should not be templated at runtime. There's a solution for templates and for tasks. Let's start with templates, there's '{% raw %}' and '{% endraw %}'. Here's an example: {% raw %} a string that won't be templated and contains a {{ var }} {% endraw %} This will result in the following file content: a string that won't be templated and contains a {{ var }} Now for playbooks, you can use '!unsafe' in front of the line of text. This will make ansible ignore the jinja variable definition and treat it as text.  Now to the problem that spawned this post: creating a mail notification with custom messages in AWX. Here's an example playbook: --- - hosts: localhost become: false delegate_to: localhost collections: - awx.awx tasks: - name: create mail notification awx.a...

First of all, make sure you have a working backup since this method is potentially destructive! I am not responsible for any data loss or broken configurations in your environment. I've run into a situation where multiple tasks (~8.000) tasks were stuck in execution over the course of a week. Canceling them by hand will take me forever and the WebUI is non-responsive when selecting this amount of tasks, so foreman-rake to the rescue it is. Start by finding out the label of the process you are going to stop: [root@katello ~]# su - postgres -c "psql -d foreman -c 'select label,state,result from foreman_tasks_tasks where state = '\''running'\'' and result = '\''pending'\''';" For me the label was 'Actions::RemoteExecution::RunHostJob'. Here're all the details I was searching for using foreman-rake: label: 'Actions::RemoteExecution::RunHostJob' state: 'running' or 'pending' result: ...