Archy's Blog

While upgrading my Foreman+Katello Server to the latest version , I've encountered the following issue: [archy@katello01 ~]$ sudo foreman-installer --scenario katello 2025-04-20 11:25:08 [NOTICE] [root] Loading installer configuration. This will take some time. 2025-04-20 11:25:12 [NOTICE] [root] Running installer with log based terminal output at level NOTICE. 2025-04-20 11:25:12 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions. 2025-04-20 11:25:14 [NOTICE] [checks] System checks passed 2025-04-20 11:25:21 [NOTICE] [pre] The Foreman database foreman does not exist. 2025-04-20 11:25:21 [NOTICE] [configure] Starting system configuration. 2025-04-20 11:25:31 [NOTICE] [configure] 250 configuration steps out of 1939 steps complete. 2025-04-20 11:25:34 [NOTICE] [configure] 500 configuration steps out of 1940 steps complete. 2025-04-20 11:25:42 [NOTICE] [configure] 1000 configu...

      NOTE: this guide exists for Upgrading from v3.12 to v3.13 as well -->  here With the current version, the  official documentation  is quite good and can be referenced. I would recommend executing all of these commands in a tmux session so that your session will remain on the server in case anything happens to your workstation. Start by checking for running tasks that would prohibit an update: [root@katello01 ~]# foreman-rake katello:upgrade_check Next, update the katello host and reboot if yum tells you to: [root@katello01 ~]# dnf -4 --refresh upgrade [root@katello01 ~]# dnf needs-restarting -r If there were any updates to foreman-related packages, make sure foreman is in a consistent state: [root@katello01 ~]# foreman-maintain service stop [root@katello01 ~]# foreman-installer --scenario katello When the katello services have started again, upgrade the release-rpms: [root@katello01 ~]# dnf -y --refresh upgrade https://yum.theforema...

The prerequisites include configuring a local registry mirror for OpenShift images and ensuring a functional OpenShift cluster with administrative privileges.  Optionally , if you are using a custom certificate authority (CA), a ConfigMap containing the CA certificate must be configured in the 'openshift-config' namespace under the 'updateservice-registry' key. For example: [archy@helper01 ~]$ oc -n openshift-config get configmap/custom-ca -o yaml apiVersion: v1 kind: ConfigMap metadata: namespace: openshift-config name: custom-ca spec: ca-bundle.crt: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- updateservice-registry: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- You can add additional trusted CAs by updating the 'image.config.openshift.io/cluster' resource: [archy@helper01 ~]$ oc patch image.config.openshift.io/cluster -p '{"spec":{"additionalTruste...

Following a server reboot, I encountered an issue where the rpcbind.service failed to start. Manually restarting the service resulted in subsequent login failures, as indicated by the following error: System is booting up. Unprivileged users are not permitted to log in yet. Please come back later. For technical details, see pam_nologin(8).   Here's a record of how I approached debugging this issue. I'm sharing it in case my experience can help others learn and troubleshoot similar problems. The server is joined to FreeIPA and HBAC rules permit access, as confirmed by the following 'ipa hbactest' output: [root@admin03 ~]# ipa hbactest --user archy --host admin03.archyslife.lan --service sshd | grep -viIE 'not' -------------------- Access granted: True -------------------- Matched rules: allow-admin-users-admin-hosts Reviewing the /var/log/secure log showed a PAM account permission issue that was blocking user logins: Feb 17 18:23:26 admin03 ss...

Ansible Automation Platform's Execution Environments offer a robust and efficient solution for managing automation workflows. By encapsulating playbook dependencies within a containerized environment, these environments ensure consistency and reproducibility across deployments, while also preventing conflicts with the underlying system's packages. This approach not only streamlines automation processes but also enhances security by isolating playbook execution, mitigating potential risks associated with dependency conflicts or vulnerabilities. Before installing ansible-builder, confirm that the system has access to the required repositories. Once confirmed, proceed with the installation of ansible-builder and podman by running the command below: [archy@ansible ~]$ sudo dnf -y --refresh install ansible-builder podman Now, we'll create the execution-environment.yml file that defines the configuration for our Ansible Execution Environment. To keep the home-directory organiz...

OpenShift offers the capability to leverage a Directory Service as an authentication source for user lookup. However, it lacks native functionality for automatic group synchronization into its Identity Management system. To achieve group syncing, we will implement a workaround solution. This involves deploying a cronjob that synchronizes groups from the source LDAP (FreeIPA in this instance) to OpenShift's OAuth service. Start by creating a dedicated namespace to encapsulate all the resources required for the automated synchronization of LDAP groups with OpenShift. $ cat << EOF > 0-namespace.yml --- apiVersion: v1 kind: Namespace metadata: name: ldap-group-sync ... EOF $ oc apply -f 0-namespace.yml Given the use of LDAPS with certificates signed by a custom Certificate Authority (CA), it is necessary to create: A ConfigMap to store the CA certificate. A Secret to securely store the password for the service account used for LDAP lookups." $ o...

      NOTE: this guide exists for Upgrading from v3.11 to v3.12 as well -->  here Additionally, ensure the successful completion of the upgrade/migration to an EL9 host. Refer to this post for guidance on upgrading from EL8 to EL9 using Leapp:  Foreman - upgrade from el8 to el9 using leapp . With the current version, the  official documentation  is quite good and can be referenced. I would recommend executing all of these commands in a tmux session so that your session will remain on the server in case anything happens to your workstation. Start by checking for running tasks that would prohibit an update: [root@katello01 ~]# foreman-rake katello:upgrade_check Next, update the katello host and reboot if yum tells you to: [root@katello01 ~]# dnf -4 --refresh upgrade [root@katello01 ~]# dnf needs-restarting -r If there were any updates to foreman-related packages, make sure foreman is in a consistent state: [root@katello01 ~]# foreman-maintai...

Creating users is typically straightforward as the documentation for the required Ansible modules is comprehensive and easy to navigate. However, working with dictionaries instead of lists can introduce some additional complexity.  For example, let's assume the following dictionary structure is given: usergroups: group1: gid: 10001 name: group1 group2: gid: 10002 name: group2 group3: gid: 10003 name: group3 group4: gid: 10004 name: group4 users: user1: uid: 1985 name: user1 groups: - group1 - group2 sshkeys: - ssh-ed25519 AAAA0 - ssh-ed25519 AAAA1 user2: uid: 1986 name: user2 groups: - group4 - group1 sshkeys: - ecdsa-sha2-nistp384 AAAA0 Ansible provides the dict2items filter which transforms a dictionary into a list of key-value pairs. This transformation allows you to ite...

Foreman has supported el9 for several releases. With Foreman 3.12 and Katello 4.14 being the last supported versions on el8, it's time to upgrade. Upgrading to el9 will provide access to future upgrades, streamline the verisons used in my infrastructure and the latest security and performance improvements. I'll be upgrading my Server using leapp again just like I did previously. The procedure is outlined quite well in the documentation but here's my own take on documenting the procedure. NOTE: before continuing, please ensure you have a way to roll back any changes in case anything goes wrong. First, apply the latest patches for all packages: [root@katello01 ~]# dnf -4y --refresh update Check if a reboot is required: [root@katello01 ~]# dnf needs-restarting -r Perform a reboot if system libraries and packages have been upgraded: [root@katello01 ~]# systemctl reboot Once the server is back online, we can begin with the upgrade. To facilitate the transition from EL8 ...