Archy's Blog

There can be weird cases where the domain.tld is being appended twice in the ansible facts. This will most likely be caused by resolving the PTR Record in your DNS Zone which might append the zone twice if you forget the '.' at the end of the hostname: [22:43:26 - archy@stealth-falcon ~]$ ipa dnsrecord-show 10.31.172.in-addr.arpa 13 Record name: 13 PTR record: elk02.archyslife.lan.archyslife.lan. I'm using FreeIPA, so the fix is fairly straight forward: [23:44:47 - archy@stealth-falcon ~]$ ipa dnsrecord-mod 10.31.172.in-addr.arpa 13 --ptr-rec 'elk02.archyslife.lan.' Now, verify that the a and ptr records resolve as expected: [23:44:58 - archy@stealth-falcon ~]$ dig +short -x $(dig +short a elk02.archyslife.lan) elk02.archyslife.lan. TL,DR:  it was dns Feel free to comment and / or suggest a topic.

When upgrading your infrastructure, the decommissioning of servers is a common task. When it comes to FreeIPA, some preparations have to be accounted for. In order to ensure proper replication, first, you'll have to remove the server from FreeIPA: [archy@ipa02 ~]$ kinit archy [archy@ipa02 ~]$ ipa server-del ipa01.archyslife.lan --ignore-topology-disconnect Now that the server is removed including all the replication agreements, you can go ahead and uninstall the service from the server. [root@ipa01 ~]# ipa-server-install --uninstall Keep in mind that the server will need a reboot after completion. The server should now have been cleanly removed from your setup. Freel free to comment and / or suggest a topic.

This error will most likely be caused when the ipa-server that originally issued the keytab is not reachable. In my case, the server has been decommissioned so the way to go is fix-forward. First, get a new keytab from one of your ipa-replicas: [root@katello ~]# mv /etc/foreman-proxy/freeipa.keytab /etc/foreman-proxy/freeipa.keytab.old [root@katello ~]# kinit admin [root@katello ~]# ipa-getkeytab -s ipa02.archyslife.lan -p katello-realm@ARCHYSLIFE.LAN -k /etc/foreman-proxy/freeipa.keytab With the keytab present, make sure the SELinux context is correct: [root@katello ~]# restorecon -v /etc/foreman-proxy/freeipa.keytab Now re-run the installer to ensure foreman's consistency: [root@katello ~]# foreman-installer --scenario katello Creating a host with the associated realm should now work. Feel free to comment and / or suggest a topic.

SSSD-KCM is a service tool for managing kerberos caches obtained from pam_sso but it can sometimes cause huge cpu-loads (90%+). I'm not sure on why exactly this happens but I think it has something to do with the secrets.ldb (/var/lib/sss/secrets/secrets.ldb) mismatching the current dyanmic-db or cache. Anyway, the solution is fairly simple and takes less than a minute. First, stop both sssd-services: [root@server ~]# systemctl stop sssd.service sssd-kcm.socket Now, make a backup of the secrets.ldb, you know just in case: [root@server ~]# cp -r /var/lib/sss/secrets /var/lib/sss/secrets.bak [root@server ~]# rm -rf /var/lib/sss/secrets Start the sssd services again: [root@server ~]# systemctl start sssd.service sssd-kcm.socket The 'sssd_kcm' process should not be causing any huge cpu loads anymore. Feel free to comment and / or suggest a topic.

I've encountered this error in a FreeIPA - AD Trust environment along with this error: "pam_sss(sshd_auth): received for user: 6 (Permission denied)" where users could log in using ssh with GSSAPIAuthentication and PubkeyAuthentication but logins with passwords were rejected. I've found that removing the dynamic-db files, clearing the cache, and restarting sssd worked for me: [root@server ~]# systemctl stop sssd.service [root@server ~]# rm -rf /var/lib/sss/db/* [root@server ~]# sss_cache -E [root@server ~]# systemctl start sssd.service I'm not really sure what exactly is causing this error but I think it might have to do with password-changes and therefore invalid caches but this is just a wild guess to the best of my knowledge. Feel free to comment and / or suggest a topic.

 I've seen this frequently now that the restore process is being killed for, from the first glance at it, no apparent reason. Checking journalctl, the problem becomes obvious: Out of memory: Killed process $PID (mongorestore) ... So the restore process requires huge amounts of memory in order to restore the Database. I've had the same problem with mongoimport and I found the solution to be either add more RAM to the machine add a large swap partition / file to the system Since RAM is probably more 'expensive' I'd suggest going with the second option and just adding another disk if necessary. In my case, the total size of the database was ~5G and after some experimenting I found out that I had to have atleast 18G of RAM available for the mongorestore / mongoimport to go through. Let's add a disk to the server on the hypervisor [root@hyv01 ~]# virsh vol-create-as --pool vm-data --name test.qcow2 --capacity 32G --format qcow2 --prealloc-metadata [root@hyv01 ...

OEMs can sometimes ship very flakey RPMs when it comes to installing some servicing software and they won't install correctly. I've recently had this experience on a centos host running some OEM-Specific software. First, inspect the installer script for the rpm. Usually, it's fairly obvious why any error is returned: [root@server ~]# rpm -qp --scripts RepServer-2.01.03.x86_64.rpm | less In my case, the script was failing due to some specific versions of the libcurl, libcrypto and libssl libraries not being present. So, ln to the rescue? [root@server ~]# ln -s /usr/lib64/libssl.so.1.1.1g /usr/lib64/libssl.so.10 [root@server ~]# ln -s /usr/lib64/libssl.so.1.1.1g /usr/lib64/libssl.so.1.0.1e [root@server ~]# ln -s /usr/lib64/libcrypto.so.1.1.1g /usr/lib64/libcrypto.so.10 [root@server ~]# ln -s /usr/lib64/libcrypto.so.1.1.1g /usr/lib64/libcrypto.so.1.0.1e [root@server ~]# ln -s /usr/lib64/libcurl.so.4.5.0 /usr/lib64/libcurl.so.4.1.1 Now, let's try to instal...

Usually, sshfp records are being generated if the host is being joined to the ipa-domain either by hand or by a provisioning workflow provided by foreman for example. If you have to regenerate them, however, they won't match and you'll get the warning that the keys don't match, rightfully so.  In order to fix this, you just need to modify the host with the current public keys and pass the '--updatedns'-option to the ipa command. First, let's make sure that all the ssh-host keys are newly generated: [root@server ~]# find /etc/ssh/ -type f -name '*ssh_host_*_key.pub' -delete [root@server ~]# ssh-keygen -A The keys won't match now and if you log out and in again, you'll get the warning I mentioned earlier. Now, let's regenerate the records. The ssh-keygen will also create a DSA key which is strongly advised against using since they're mathematically broken. So, we're just going to print out the ECDSA, ED25519, and RSA keys: [root@...

A neat little snippet when migrating datasets between servers or creating archives to remote servers when disk space is limited. Here are some examples I've used extensively when migrating some RHEL7 Servers to RHEL8: Creating archive locally and push to remote server: [root@kvm ~]# tar -cvpJf - /srv/kvm/vm-images | ssh root@nas.archyslife.lan 'cat > /var/backup/vm-images.tar.xz' Create archive remotely and pull to local server: [root@nas ~]# ssh archy@kvm.archyslife.lan 'tar -cvpJf - /srv/kvm/vm-images' > /var/backup/vm-images.tar.xz Pull archive from remote server and extract locally: [archy@kvm ~]$ ssh archy@nas.archyslife.lan 'cat /var/backup/vm-images.tar.xz' | tar -xvJf - -C / Push archive from local server to  [archy@nas ~]$ cat /var/backup/vm-images.tar.xz | ssh archy@kvm.archyslife.lan 'tar -xvpJf - -C /' Feel free to comment and / or suggest a topic.

I've just migrated 2 AWX Hosts from RHEL7 to RHEL8 and I was using the docker-compose method. This obviously requires docker-compose which can be installed using pip3 in RHEL8 but when using 'latest' as version, you'll encounter an error like 'Rust compiler not installed'. This error is caused by the latest version of the cryptography module in pip's repositories. I've found these versions to be usable with RHEL8: cffi==1.12.0 cryptography==3.3.1 docker-compose==1.28.0 These modules can be installed using pip: [archy@awx ~]# sudo pip3 install cffi==1.12.0 cryptography==3.3.1 docker-compose==1.28.0 A corresponding ansible snippet could look like this: - name: install docker-compose if os_major is 8 pip: name: "{{ item }}" state: present loop: - cffi==1.12.0 - cryptography==3.3.1 - docker-compose==1.28.0 when: ansible_facts['distribution_major_version'] == '8' tags: ...

This error occurred for me after updating the dhcp-server if it's integrated into foreman using either the 'isc' or 'remote_isc' module. The fix was very simple since all I had to do was making sure the permissions are set accordingly and restart dhcpd. [archy@dhcpd ~]$ sudo chmod 755 /etc/dhcpd [archy@dhcpd ~]$ sudo systemctl restart dhcpd.service Foreman should now be able to assign addresses again. Feel free to comment and / or suggest a topic.

With tmux you can easily share a session between mutliple users. This is very handy when debugging a problem for example or 'working with 4 eyes' on something. In order to do that, first launch tmux with specifying a socket (-S) and give the session a name: [root@server ~]# tmux -S /tmp/shared new -s shared_session In order to make it so that anyone else can connect, make sure the permissions are rw for both members: [root@server ~]# chmod 666 /tmp/shared As the second user, attach to tmux specifying the socket and the session name: [archy@server ~]$ tmux -S /tmp/shared attach -t shared_session The output and input are now shared between both users. Feel free to comment and / or suggest a topic.

I prefer to use the respective config files for services in order to configure them. An example here is ssh where if you configure for example Ciphers, KexAlgorithms, and MACs in the sshd_config it most likely won't take effect. RHEL8 has switched to system-wide crypto policies which also affect sshd.  To make sshd ignore the crypto policies, uncomment the 'CRYPO_POLICY=' line in /etc/sysconfig/sshd: [archy@server ~]$ sudo sed -i 's/^#\ CRYPTO_POLICY=/CRYPTO_POLICY=/g' /etc/sysconfig/sshd Restart sshd: [archy@server ~]$ sudo systemctl restart sshd.service Check with nmap to see if the settings have been applied: [archy@server ~]$ nmap -sV -Pn -p 22 -open -script ssh2-enum-algos 127.0.0.1 Starting Nmap 7.80 ( https://nmap.org ) at 2021-01-26 19:40 CET Nmap scan report for 127.0.0.1 Host is up (0.00096s latency). PORT STATE SERVICE VERSION 22/tcp open ssh OpenSSH 8.0 (protocol 2.0) | ssh2-enum-algos: | kex_algorithms: (1) | curv...

Creating a floating IP between 2 hosts is very handy when dealing with High Availability in dependency of one or more services. I'll use a floating IP between 2 IPA Servers in Order to create an HA-DNS Endpoint so that servers can be upgraded independently. Here's a short description: ipa01.archyslife.lan:   - ip: 172.31.10.5   - service: named-pkcs11   - keepalived-state: MASTER ipa02.archyslife.lan   - ip: 172.31.10.6   - service: named-pkcs11   - keepalived-state: BACKUP First of all, install the 'keepalived' package on all servers: [archy@ipa01 ~]$ sudo yum -y install keepalived [archy@ipa02 ~]$ sudo yum -y install keepalived Now, create the configuration for the main Instance first: [archy@ipa01 ~]$ sudo vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { notification_email { admins@archyslife.lan } notification_email_from root@ipa01.archyslife.lan smtp_server 127.0.0.1 smtp...

 With a recent announcement from the kubernetes project, docker will no longer be supported as of kubernetes version 1.22 (end of 2021). What this means in practice is that you'll have to move to some other container engine such as CRI-O or containerd.io. Since containerd (containerd.io rpm) is already included with the docker repos, I'll use these since I've already have them synced in katello. First up, this is a very simple 3-node setup, I will not cover high availability with etcd or shared storage. Here's the nodes I will use: kube-master01.archyslife.lan - role: master - ip: 172.31.10.123 - vcpus: 2 - ram: 4g kube-worker01.archyslife.lan - worker - role: worker - ip: 172.31.10.125 - vcpus: 2 - ram: 4g kube-worker02.archyslife.lan - worker - role: worker - ip: 172.31.10.126 - vcpus: 2 - ram: 4g Each of them will be running CentOS 7 ( reason  and I also have not yet migrated to 8-stream since I want to wait on the announcements from Red Hat and releases of Rocky Lin...

Encrypting your drives is becoming more and more mandatory especially for mobile devices and storage media such as USB-Sticks. Cryptsetup is 'an easy to use'-tool in Linux which can take care of encrypting storage devices. First of all, initialize the partition using cryptsetup: # cryptsetup luksFormat /dev/sda1 WARNING! ======== This will overwrite data on /dev/sda1 irrevocably. Are you sure? (Type 'yes' in capital letters): YES Enter passphrase for /dev/sda1: Verify passphrase: Now that the partition has been initialized, open if with an assigned device-mapper name 'tempusb': # cryptsetup open /dev/sda1 tempusb Enter passphrase for /dev/sda1: Now that the partition is opened, you can create a filesystem on the partition: # mkfs.xfs /dev/mapper/tempusb Now that the filesystem is created, you can mount it and write to it. For the sake of an example, I'll copy the /home/ directory. # mount /dev/mapper/tempusb /mnt # rsync -vrlp...

I've found this error to be hard to pin point what exactly is going wrong here since it just says 'unable to retrieve unused IP'. If you are on a roblox dhcp, there's solutions available in the foreman community but since in this setup I was using dhcpd (isc-dhcp), they were not applicable. Here's the complete error message from the logs: 2020-12-04T11:27:31 [W|app|83514b3e] Failed to fetch a free IP from proxy katello.archyslife.lan (https://katello.archyslife.lan:9090): ERF12-8202 [ProxyAPI::ProxyException]: Unable to retrieve unused IP ([RestClient::NotFound]: 404 Not Found) for Capsule https://katello.archyslife.lan:9090/dhcp TL;DR, here's the solution that worked for me: [archy@dhcpd ~]$ sudo chmod 755 /etc/dhcp [archy@katello ~]$ sudo foreman-maintain service restart The cause was most likely an update and since my environment is on CentOS which in turn means rpm packages.  When updating .rpm packages, they can trigger a script which probably c...

By default, the freeipa-server installer will configure bind automatically into a working state. From time to time, it might be necessary to adjust some dynamic values such as tracing for debugging purposes.  Bind ships with a convenient tool called 'rndc' to control bind dynamically. However, this is not enabled by default so there's some work to do. In order to activate the remote control, you'll have to first create a rndc-key: [archy@ipa01 ~]$ sudo rndc-confgen -a -b 512 -c /etc/control-key -k control-key [archy@ipa01 ~]$ sudo chmod 644 /etc/control-key [archy@ipa01 ~]$ sudo restorecon -Rv /etc/control-key With the key created, create a control section in the named.conf to allow the key to control the named service. Open the named.conf file with an editor of your choice and add these lines: [archy@ipa01 ~]$ sudo vim /etc/named.conf include "/etc/control-key"; controls { inet 127.0.0.1 port 953 allow { 127.0.0.1; } keys { "c...

After updating to RHEL 8.3, I've noticed that DNS-Queries return nothing when using the integrated named with FreeIPA. [archy@web01 ~]$ dig +short a katello.archyslife.lan @172.31.10.5 [archy@web01 ~]$ The clients are being denied as can be seen by checking the logs in /var/named/data/named.run: 30-Nov-2020 06:36:49.508 client @0x7f0ca41539f0 172.31.10.111#46891 (web01.archyslife.lan): query (cache) 'web01.archyslife.lan/A/IN' denied This can be resolved by adding a query-policy to the end of the file /etc/named/ipa-options-ext.conf. [archy@ipa01 ~]$ sudo vim /etc/named/ipa-options-ext.conf allow-query { any; }; Check your config with named-checkconf to make sure there are no syntax errors. [archy@ipa01 ~]$ sudo named-checkconf; if [ $? -eq 0 ]; then echo 'all good'; else echo 'mistakes have been made'; fi If everything is okay, go ahead and restart the services. [archy@ipa01 ~]$ sudo ipactl restart Let's try again to check if it...