Skip to main content

Foreman - upgrade from el8 to el9 using leapp

Foreman has supported el9 for several releases. With Foreman 3.12 and Katello 4.14 being the last supported versions on el8, it's time to upgrade. Upgrading to el9 will provide access to future upgrades, streamline the verisons used in my infrastructure and the latest security and performance improvements. I'll be upgrading my Server using leapp again just like I did previously. The procedure is outlined quite well in the documentation but here's my own take on documenting the procedure.

NOTE: before continuing, please ensure you have a way to roll back any changes in case anything goes wrong.

First, apply the latest patches for all packages:

 [root@katello01 ~]# dnf -4y --refresh update

Check if a reboot is required:

 [root@katello01 ~]# dnf needs-restarting -r

Perform a reboot if system libraries and packages have been upgraded:

 [root@katello01 ~]# systemctl reboot

Once the server is back online, we can begin with the upgrade. To facilitate the transition from EL8 to EL9, we will enable the foreman/leapp copr repository, which provides the necessary tools and packages for the upgrade process. Then, we will install the required packages:

 [root@katello01 ~]# dnf -4y --refresh copr enable @theforeman/leapp  
 [root@katello01 ~]# dnf -4y --refresh install leapp leapp-upgrade-el8toel9

I'll be upgrading from Almalinux 8 to Almalinux 9, so I'll have to install the 'leapp-data-almalinux' package:

 [root@katello01 ~]# dnf -4y --refresh install leapp-data-almalinux

With all the packages installed, we'll have to append the Foreman-Repos to the leapp upgrade repo file:

 [root@katello01 ~]# vim /etc/leapp/files/leapp_upgrade_repositories.repo  

 [leapp-foreman]  
 name=Foreman 3.12  
 baseurl=https://yum.theforeman.org/releases/3.12/el9/$basearch  
 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-foreman  
 enabled=1  
 gpgcheck=1  
 
 [leapp-foreman-plugins]  
 name=Foreman plugins 3.12  
 baseurl=https://yum.theforeman.org/plugins/3.12/el9/$basearch  
 enabled=1  
 gpgcheck=0  
 
 [leapp-katello]  
 name=Katello 4.14  
 baseurl=https://yum.theforeman.org/katello/4.14/katello/el9/$basearch/  
 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-foreman  
 enabled=1  
 gpgcheck=1  
 
 [leapp-candlepin]  
 name=Candlepin: an open source entitlement management system.  
 baseurl=https://yum.theforeman.org/candlepin/4.4/el9/$basearch/  
 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-candlepin  
 enabled=1  
 gpgcheck=1  
 
 [leapp-pulpcore]  
 name=pulpcore: Fetch, Upload, Organize, and Distribute Software Packages.  
 baseurl=https://yum.theforeman.org/pulpcore/3.49/el9/$basearch/  
 gpgkey=https://yum.theforeman.org/pulpcore/3.49/GPG-RPM-KEY-pulpcore  
 enabled=1  
 gpgcheck=1  
 
 [leapp-puppet8]  
 name=Puppet 8 Repository el 9 - $basearch  
 baseurl=http://yum.puppetlabs.com/puppet8/el/9/$basearch  
 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-2025-04-06-puppet8-release  
 enabled=1  
 gpgcheck=1

That's the leapp configuration done. Now ensure all Foreman services are stopped:

 [root@katello01 ~]# foreman-maintain service stop

I'm using a nfs share to files to foreman for the dhcp-proxy feature, so I'll have to unmount them and comment them out in '/etc/fstab':

 [root@katello01 ~]# umount /mnt/nfs/etc/dhcp /mnt/nfs/var/lib/dhcpd  
 [root@katello01 ~]# vim /etc/fstab  

 # satellite capsule stuff  
 # infra01.archyslife.lan:/exports/etc/dhcp /mnt/nfs/etc/dhcp nfs ro,vers=3,auto,nosharecache,context="system_u:object_r:dhcp_etc_t:s0" 0 0  
 # infra01.archyslife.lan:/exports/var/lib/dhcpd /mnt/nfs/var/lib/dhcpd nfs ro,vers=3,auto,nosharecache,context="system_u:object_r:dhcpd_state_t:s0" 0 0

Since I've upgraded my installation over time, there are still some packages left over from previous versions of the foreman and katello packages that will cause dependency issues down the road. We'll have to remove them beforehand:

 [root@katello01 ~]# dnf -4y autoremove  
 [root@katello01 ~]# dnf -4y remove python39

The upgrade using leapp will cause a lot of open file handles, so we'll have to increase the available file handles using 'ulimit' for the current shell session:

 [root@katello01 ~]# ulimit -n 65536

Now, let's run the preupgrade and check for inhibitors:

 [root@katello01 ~]# leapp preupgrade

The report shows no inhibitors, so let's run the upgrade:

 [root@katello01 ~]# leapp upgrade

This command will take a few minutes to complete since it will download a lot of packages and create a initramfs that will take care of the upgrade. That also means that there's another reboot required:

 [root@katello01 ~]# systemctl reboot

The system will now reboot into the newly generated initramfs and perform the upgrade from el8 to el9.

Progress can be monitored using 'journalctl' and following the service 'leapp-resume.service':

 [root@katello01 ~]# journalctl -b -e -f -u leapp-resume.service

In my case, there was an error because of duplicate entries of puppet facts. The fix is documented by Red Hat in KB6998041:

 [root@katello01 ~]# foreman-maintain service stop --exclude postgresql  
 [root@katello01 ~]# cat << EOF | foreman-rake console  
 conf.echo = false  
 FactName.unscoped.all.pluck(:type).uniq.each do |type|  
  type.constantize.unscoped.group_by(&:name).select{|k,v| v.size > 1}.values.each do |dup_names|  
   existing = dup_names[0]  
   dup_name_ids = dup_names[1..-1].map(&:id)  
   p "Found duplicate fact names #{existing.type} #{existing.name} ids #{dup_name_ids}"  
   values = FactValue.unscoped.where(fact_name_id: dup_name_ids)  
   p "Merging #{values.size} fact values."  
   values.in_batches.update_all(fact_name_id: existing.id)  
   p "Deleting duplicate fact names."  
   FactName.unscoped.where(id: dup_name_ids).destroy_all  
  end  
 end  
 EOF

Now, re-run the installer to ensure a consistent system state:

 [root@katello01 ~]# foreman-installer --scenario katello

After the installer has run through without an issue, continue to enable selinux and reboot once more:

 [root@katello01 ~]# sed -i 's/SELINUX=permissive/SELINUX=enforcing/g' /etc/selinux/config  
 [root@katello01 ~]# systemctl reboot

After the server has come up again with SELinux enabled, stop the foreman services and run the installer once last time for this upgrade:

 [root@katello01 ~]# foreman-maintain service stop  
 [root@katello01 ~]# foreman-installer --scenario katello

After completion, check the health of the system:

 [root@katello01 ~]# foreman-maintain service status  
 [root@katello01 ~]# foreman-maintain health check

Okay, everything seems healthy and stable so far. We can now move to cleaning up the system from old resources. First, clean up any old packages that are left over from el8. There's a good article on that here: Performing post-upgrade tasks on the RHEL 9 system. It essentially boils down to these commands:

 [root@katello01 ~]# rpm -qa | grep -e '\.el[78]' | grep -vE '^(gpg-pubkey|libmodulemd|katello-ca-consumer)' | sort  
 [root@katello01 ~]# dnf -y remove $(rpm -qa | grep -iIE '\.el[78]' | grep -viIE 'gpg-pubkey|libmodulemd|katello-ca-consumer')  
 [root@katello01 ~]# dnf -y remove leapp-deps-el9 leapp-repository-deps-el9  
 [root@katello01 ~]# rm -rf /var/log/leapp /root/tmp_leapp_py3 /var/lib/leapp

The only thing left to do now is replacing the old rescue-kernel and initramfs with the current versions. Again, theres a good resource available online here: Performing post-upgrade tasks on the RHEL 9 system. Start by removing the old initramfs and rescue-kernels:

 [root@katello01 ~]# rm /boot/vmlinuz-*rescue* /boot/initramfs-*rescue*

Then, regenerate both of them and let dracut create entries in the bootmenu:

 [root@katello01 ~]# /usr/lib/kernel/install.d/51-dracut-rescue.install add "$(uname -r)" /boot "/boot/vmlinuz-$(uname -r)"

That's it, the system is now upgraded from el8 to el9 with a working Foreman/Katello installation.

Feel free to comment and / or suggest a topic.

Labels:

Comments

Post a Comment

Popular posts from this blog

Dynamic DNS with BIND and ISC-DHCP

I personally prefer to work with hostnames instead of ip-addresses. If you have anything like freeipa or active directory, it will do that for you by registering the client you added to your realm to the managed dns and edit the records dynamically. We can achieve the same goal with just bind and isc-dhcp. I'll use a raspberry pi with raspbian 9 for this setup. So here is a quick tutorial on how to configure the isc-dhcp-server to dynamically update bind. First set a static ip to your server. [archy@ddns ~]$ sudo vim /etc/network/interfaces # interfaces(5) file used by ifup(8) and ifdown(8) # Please note that this file is written to be used with dhcpcd # For static IP, consult /etc/dhcpcd.conf and 'man dhcpcd.conf' # Include files from /etc/network/interfaces.d: source-directory /etc/network/interfaces.d auto eth0 iface eth0 inet static address 172.31.30.5 network 172.31.30.0 broadcast 172.31.30.255 netmask 255.255.255.0 ...
Post a Comment
Read more

LACP-Teaming on CentOS 7 / RHEL 7

What is teaming? Teaming or LACP (802.3ad) is a technique used to bond together multiple interfaces to achieve higher combined bandwith. NOTE: every clients speed can only be as high as the single link speed of one of the members. That means, if the interfaces I use in the bond have 1 Gigabit, every client will only have a maximum speed of 1 Gigabit. The advantage of teaming is, that it can handle multiple connections with 1 Gigabit. How many connections depends on the amount of your network cards. I'm using 2 network cards for this team on my server. That means I can handle 2 Gigabit connections at full rate on my server provided the rest of the hardware can deliver that speed. There also exists 'Bonding' in the Linux world. They both do the same in theory but  for a detailed comparison check out this  article about teaming in RHEL7 . To create a teaming-interface, we will first have to remove all the interface configurations we've done on the (soon to be) sla...
8 comments
Read more

Push logs and data into elasticsearch - Part 2 Mikrotik Logs

This is only about the setup of different logging, one being done with Filebeat and the other being done with sending logging to a dedicated port opened in Logstash using the TCP / UDP Inputs. Prerequesites: You'll need a working Elasticsearch Cluster with Logstash and Kibana. Start by getting the Log Data you want to structure parsed correctly. Mikrotik Logs are a bit difficult since they show you Data in the interface which is already enriched with Time / Date. That means a message that the remote logging will send to Logstash will look like this: firewall,info forward: in:lan out:wan, src-mac aa:bb:cc:dd:ee:ff, proto UDP, 172.31.100.154:57061->109.164.113.231:443, len 76 You can check them in the grok debugger and create your own filters and mapping. The following is my example which might not fit your needs. Here are some custom patterns I wrote for my pattern matching: MIKROTIK_DATE \b(?:jan(?:uary)?|feb(?:ruary)?|mar(?:ch)?|apr(?:il)?|may|jun(?:e)?|jul(?...
9 comments
Read more

FreeIPA - Integrating your DHCPD dynamic Updates into IPA

I recently went over my network configuration and noticed that the dhcp-leases were not pushed into the IPA-DNS yet. So I thought, why not do it now. The setup is very similar to setting it up on a single bind instance not managed by IPA (I've already written a guide about this here ). My setup is done with the following hosts: ipa01.archyslife.lan - 172.31.0.1 inf01.archyslife.lan - 172.31.0.5 First of all, create a rndc-key: [archy@ipa01 ~]$ sudo rndc-confgen -a -b 512 This will create the following file '/etc/rndc-key' [archy@ipa01 ~]$ sudo cat /etc/rndc.key key "rndc-key" { algorithm hmac-md5; secret "secret_key_here=="; }; We also need to make named aware of the rndc-key and allow our remote dhcp server to write dns entries: [archy@ipa01 ~]$ sudo vim /etc/named.conf ... include "/etc/rndc-key"; controls { inet 172.31.0.1 port 953 allow { 172.31.0.5; } keys ...
2 comments
Read more

SSSD - Debugging PAM permission denied

Sometimes there's weird errors in IT that occur on random chance. I've had such an encounter with SSSD in combination with IPA(+AD-Trust) recently, where only sometimes, a connection to one of the IPA-Servers would fail with this error: Jul 13 13:36:42 ipa02.archyslife.lan sshd[3478]: pam_sss(sshd:account): Access denied for user runner: 4 (System error) Jul 13 13:36:42 ipa02.archyslife.lan sshd[3478]: fatal: Access denied for user runner by PAM account configuration [preauth] In my case, it was only happening sometimes when running a basic system setup role using ansible on every host in the entire environment. This way, there was no consistent pattern besides being the same host every time if it failed. First up, add the 'debug_level=X' to every section required in the /etc/sssd/sssd.conf where X is a number from 1 to 10 with 10 being the most verbose. Afterward, restart sssd and check the logs for any obvious problems. 1) If you are using local users, check the...
Post a Comment
Read more