Archy's Blog

When using conditionals in ansible, most of the time you'll use 'when' and specify which parameter should be checked. This can however result in a rather lengthy and confusing list of tasks that can be easily reduced. Here's an example of a list of tasks that create users: - name: create regular users | --password, --groups become: true loop: "{{ users }}" user: name: "{{ item.name }}" uid: "{{ item.uid }}" shell: "{{ item.shell | default('/bin/sh') }}" when: not item.groups is defined and not item.password is defined tags: - all - users - name: create regular users | --password, ++groups become: true loop: "{{ users }}" user: name: "{{ item.name }}" uid: "{{ item.uid }}" shell: "{{ item.shell | default('/bin/sh') }}" groups: "{{ item.groups }}" when: ite...

When there's a foreman present in the environment, it's wise to use it as an inventory source for your ansible deployments since it will most likely always be up to date when it comes to hosts (single source of truth).  In order to use ansible as inventory, you'll have to define a *.foreman.yml file in your inventory which will then query the foreman for hosts, groups and variables which can then be used in ansible: [archy@ansible ~/ansible]$ vim inventories/foreman/archyslife.foreman.yml plugin: foreman url: https://katello.archyslife.lan user: roadmin password: !vault | $ANSIBLE_VAULT;1.1;AES256 00000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000...

Kubernetes has the ability to automatically scale resources as required by the current load. There are some prerequirements however. With a user-provisioned kubernetes cluster, there's most likely not going to be a metrics-server deployed: [archy@kube01 ~]$ kubectl -n kube-system get deployments -o wide NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR coredns 2/2 2 2 154d coredns k8s.gcr.io/coredns/coredns:v1.8.0 k8s-app=kube-dns To deploy the metrics-server, go ahead and download the manifest from GitHub: [archy@kube01 ~]$ curl -k -L -o kubernetes/metrics-server.yml -X GET 'https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml' In order to enable the metrics-server to communicate with the worker-nodes, we'll have to add the 'kubelet-insecure-tls' option to the manifest args and then apply it: [archy@kube01 ~]$ vim kube...

Keeping the time in sync for all your hosts with multiple major versions of one operating system can be challenging. Luckily, there's the linux-system-roles.timesync-role which will take care of pretty much everything using ansible.  First, let's add the roles/requirements.yml file and install the role: [archy@ansible01 ~]$ vim roles/requirements.yml --- - src: https://github.com/linux-system-roles/timesync.git scm: git name: linux-system-roles.timesync ... [archy@ansible01 ~]$ ansible-galaxy role install -r roles/requirements.yml -p ./roles/ --force Now with the roles being present, create a playbook that will use the role and run on every host on your inventory. I'll just extend my base-playbook which will do basic configurations on all systems: [archy@ansible01 ~]$ vim deploy_base.yml - hosts: all:!ipa[0-9][0-9].archyslife.lan user: ansible-executor become: true gather_facts: true vars: timesync_ntp_servers: ...

After upgrading to RedHat Satellite 6.9.1, I've had the insights-client not uploading the inventory as expected to cloud.redhat.com. Now since this was in a corporate network, outside connections had to go through a proxy. Check the proxy settings in satellite: [archy@satellite ~]$ hammer settings show --name http_proxy [archy@satellite ~]$ hammer settings show --name http_proxy_except_list In my case, they were not set which is odd since they were set before the upgrade. Anyway, just setting them again is fast. [archy@satellite ~]$ hammer settings set --name http_proxy --value 'http://proxy.lnx.domain.tld:3128' [archy@satellite ~]$ hammer settings set --name http_proxy_except_list --value "['10.0.0.0/8', 'domain.tld']" Checking in again on one of the hosts if the connection works as expected now: [archy@server ~]$ sudo insights-client --test-connection [archy@server ~]$ sudo systemctl restart insights-client.service If the host h...

Although this error is probably rather rare, I didn't really find anything useful on the internet regarding this error so here's my shot at it I guess. The ERF42-9911 error appears when provisioning hosts and is most likely caused by the Kickstart Finish Template when calling home to the Satellite / Foreman host to tell it that the host is built. Foreman / Satellite will then cancel the build token for the host and will let it boot into the os from the installation residing on the hard drive.  If that call-home is unsuccessful, the host will stay in build mode and the Satellite / Foreman server will not cancel the build mode resulting in provisioning loops for the host. Possible cause Nr. 1: Proxy. This can happen if 'set_proxy: true' is specified in the parameters without an appropriate no_proxy parameter in place. As a result, the provisioned host will not be able to reach the Foreman / Satellite host to cancel its build mode. Possible cause Nr. 2: DHCP Fact conflicts...

Using lvm has multiple advantages especially since hard drives can be added, and pooled. But with hard drives aging, they need to be swapped out in order to guarantee proper availability. Since with lvm, drives become irrelevant, this process is very easy. First, create the physical volume: [root@server ~]# pvcreate /dev/sdc1 Now, extend the volume group: [root@server ~]# vgextend /dev/vg_backup /dev/sdc1 With the volume group extended, the physical extents can be moved: [root@server ~]# pvmove /dev/sdb1 /dev/sdc1 This process can take a while depending on the drive speeds and sizes. Once the physical extents are moved, remove the old drive from the volume group: [root@server ~]# vgreduce /dev/vg_backup /dev/sdb1 The drives are now switched and the old drive can be removed from the system. Feel free to comment and / or suggest a topic.

There can be weird cases where the domain.tld is being appended twice in the ansible facts. This will most likely be caused by resolving the PTR Record in your DNS Zone which might append the zone twice if you forget the '.' at the end of the hostname: [22:43:26 - archy@stealth-falcon ~]$ ipa dnsrecord-show 10.31.172.in-addr.arpa 13 Record name: 13 PTR record: elk02.archyslife.lan.archyslife.lan. I'm using FreeIPA, so the fix is fairly straight forward: [23:44:47 - archy@stealth-falcon ~]$ ipa dnsrecord-mod 10.31.172.in-addr.arpa 13 --ptr-rec 'elk02.archyslife.lan.' Now, verify that the a and ptr records resolve as expected: [23:44:58 - archy@stealth-falcon ~]$ dig +short -x $(dig +short a elk02.archyslife.lan) elk02.archyslife.lan. TL,DR:  it was dns Feel free to comment and / or suggest a topic.

When upgrading your infrastructure, the decommissioning of servers is a common task. When it comes to FreeIPA, some preparations have to be accounted for. In order to ensure proper replication, first, you'll have to remove the server from FreeIPA: [archy@ipa02 ~]$ kinit archy [archy@ipa02 ~]$ ipa server-del ipa01.archyslife.lan --ignore-topology-disconnect Now that the server is removed including all the replication agreements, you can go ahead and uninstall the service from the server. [root@ipa01 ~]# ipa-server-install --uninstall Keep in mind that the server will need a reboot after completion. The server should now have been cleanly removed from your setup. Freel free to comment and / or suggest a topic.

This error will most likely be caused when the ipa-server that originally issued the keytab is not reachable. In my case, the server has been decommissioned so the way to go is fix-forward. First, get a new keytab from one of your ipa-replicas: [root@katello ~]# mv /etc/foreman-proxy/freeipa.keytab /etc/foreman-proxy/freeipa.keytab.old [root@katello ~]# kinit admin [root@katello ~]# ipa-getkeytab -s ipa02.archyslife.lan -p katello-realm@ARCHYSLIFE.LAN -k /etc/foreman-proxy/freeipa.keytab With the keytab present, make sure the SELinux context is correct: [root@katello ~]# restorecon -v /etc/foreman-proxy/freeipa.keytab Now re-run the installer to ensure foreman's consistency: [root@katello ~]# foreman-installer --scenario katello Creating a host with the associated realm should now work. Feel free to comment and / or suggest a topic.

SSSD-KCM is a service tool for managing kerberos caches obtained from pam_sso but it can sometimes cause huge cpu-loads (90%+). I'm not sure on why exactly this happens but I think it has something to do with the secrets.ldb (/var/lib/sss/secrets/secrets.ldb) mismatching the current dyanmic-db or cache. Anyway, the solution is fairly simple and takes less than a minute. First, stop both sssd-services: [root@server ~]# systemctl stop sssd.service sssd-kcm.socket Now, make a backup of the secrets.ldb, you know just in case: [root@server ~]# cp -r /var/lib/sss/secrets /var/lib/sss/secrets.bak [root@server ~]# rm -rf /var/lib/sss/secrets Start the sssd services again: [root@server ~]# systemctl start sssd.service sssd-kcm.socket The 'sssd_kcm' process should not be causing any huge cpu loads anymore. Feel free to comment and / or suggest a topic.

I've encountered this error in a FreeIPA - AD Trust environment along with this error: "pam_sss(sshd_auth): received for user: 6 (Permission denied)" where users could log in using ssh with GSSAPIAuthentication and PubkeyAuthentication but logins with passwords were rejected. I've found that removing the dynamic-db files, clearing the cache, and restarting sssd worked for me: [root@server ~]# systemctl stop sssd.service [root@server ~]# rm -rf /var/lib/sss/db/* [root@server ~]# sss_cache -E [root@server ~]# systemctl start sssd.service I'm not really sure what exactly is causing this error but I think it might have to do with password-changes and therefore invalid caches but this is just a wild guess to the best of my knowledge. Feel free to comment and / or suggest a topic.

 I've seen this frequently now that the restore process is being killed for, from the first glance at it, no apparent reason. Checking journalctl, the problem becomes obvious: Out of memory: Killed process $PID (mongorestore) ... So the restore process requires huge amounts of memory in order to restore the Database. I've had the same problem with mongoimport and I found the solution to be either add more RAM to the machine add a large swap partition / file to the system Since RAM is probably more 'expensive' I'd suggest going with the second option and just adding another disk if necessary. In my case, the total size of the database was ~5G and after some experimenting I found out that I had to have atleast 18G of RAM available for the mongorestore / mongoimport to go through. Let's add a disk to the server on the hypervisor [root@hyv01 ~]# virsh vol-create-as --pool vm-data --name test.qcow2 --capacity 32G --format qcow2 --prealloc-metadata [root@hyv01 ...

OEMs can sometimes ship very flakey RPMs when it comes to installing some servicing software and they won't install correctly. I've recently had this experience on a centos host running some OEM-Specific software. First, inspect the installer script for the rpm. Usually, it's fairly obvious why any error is returned: [root@server ~]# rpm -qp --scripts RepServer-2.01.03.x86_64.rpm | less In my case, the script was failing due to some specific versions of the libcurl, libcrypto and libssl libraries not being present. So, ln to the rescue? [root@server ~]# ln -s /usr/lib64/libssl.so.1.1.1g /usr/lib64/libssl.so.10 [root@server ~]# ln -s /usr/lib64/libssl.so.1.1.1g /usr/lib64/libssl.so.1.0.1e [root@server ~]# ln -s /usr/lib64/libcrypto.so.1.1.1g /usr/lib64/libcrypto.so.10 [root@server ~]# ln -s /usr/lib64/libcrypto.so.1.1.1g /usr/lib64/libcrypto.so.1.0.1e [root@server ~]# ln -s /usr/lib64/libcurl.so.4.5.0 /usr/lib64/libcurl.so.4.1.1 Now, let's try to instal...

Usually, sshfp records are being generated if the host is being joined to the ipa-domain either by hand or by a provisioning workflow provided by foreman for example. If you have to regenerate them, however, they won't match and you'll get the warning that the keys don't match, rightfully so.  In order to fix this, you just need to modify the host with the current public keys and pass the '--updatedns'-option to the ipa command. First, let's make sure that all the ssh-host keys are newly generated: [root@server ~]# find /etc/ssh/ -type f -name '*ssh_host_*_key.pub' -delete [root@server ~]# ssh-keygen -A The keys won't match now and if you log out and in again, you'll get the warning I mentioned earlier. Now, let's regenerate the records. The ssh-keygen will also create a DSA key which is strongly advised against using since they're mathematically broken. So, we're just going to print out the ECDSA, ED25519, and RSA keys: [root@...

A neat little snippet when migrating datasets between servers or creating archives to remote servers when disk space is limited. Here are some examples I've used extensively when migrating some RHEL7 Servers to RHEL8: Creating archive locally and push to remote server: [root@kvm ~]# tar -cvpJf - /srv/kvm/vm-images | ssh root@nas.archyslife.lan 'cat > /var/backup/vm-images.tar.xz' Create archive remotely and pull to local server: [root@nas ~]# ssh archy@kvm.archyslife.lan 'tar -cvpJf - /srv/kvm/vm-images' > /var/backup/vm-images.tar.xz Pull archive from remote server and extract locally: [archy@kvm ~]$ ssh archy@nas.archyslife.lan 'cat /var/backup/vm-images.tar.xz' | tar -xvJf - -C / Push archive from local server to  [archy@nas ~]$ cat /var/backup/vm-images.tar.xz | ssh archy@kvm.archyslife.lan 'tar -xvpJf - -C /' Feel free to comment and / or suggest a topic.

I've just migrated 2 AWX Hosts from RHEL7 to RHEL8 and I was using the docker-compose method. This obviously requires docker-compose which can be installed using pip3 in RHEL8 but when using 'latest' as version, you'll encounter an error like 'Rust compiler not installed'. This error is caused by the latest version of the cryptography module in pip's repositories. I've found these versions to be usable with RHEL8: cffi==1.12.0 cryptography==3.3.1 docker-compose==1.28.0 These modules can be installed using pip: [archy@awx ~]# sudo pip3 install cffi==1.12.0 cryptography==3.3.1 docker-compose==1.28.0 A corresponding ansible snippet could look like this: - name: install docker-compose if os_major is 8 pip: name: "{{ item }}" state: present loop: - cffi==1.12.0 - cryptography==3.3.1 - docker-compose==1.28.0 when: ansible_facts['distribution_major_version'] == '8' tags: ...

This error occurred for me after updating the dhcp-server if it's integrated into foreman using either the 'isc' or 'remote_isc' module. The fix was very simple since all I had to do was making sure the permissions are set accordingly and restart dhcpd. [archy@dhcpd ~]$ sudo chmod 755 /etc/dhcpd [archy@dhcpd ~]$ sudo systemctl restart dhcpd.service Foreman should now be able to assign addresses again. Feel free to comment and / or suggest a topic.

With tmux you can easily share a session between mutliple users. This is very handy when debugging a problem for example or 'working with 4 eyes' on something. In order to do that, first launch tmux with specifying a socket (-S) and give the session a name: [root@server ~]# tmux -S /tmp/shared new -s shared_session In order to make it so that anyone else can connect, make sure the permissions are rw for both members: [root@server ~]# chmod 666 /tmp/shared As the second user, attach to tmux specifying the socket and the session name: [archy@server ~]$ tmux -S /tmp/shared attach -t shared_session The output and input are now shared between both users. Feel free to comment and / or suggest a topic.

I prefer to use the respective config files for services in order to configure them. An example here is ssh where if you configure for example Ciphers, KexAlgorithms, and MACs in the sshd_config it most likely won't take effect. RHEL8 has switched to system-wide crypto policies which also affect sshd.  To make sshd ignore the crypto policies, uncomment the 'CRYPO_POLICY=' line in /etc/sysconfig/sshd: [archy@server ~]$ sudo sed -i 's/^#\ CRYPTO_POLICY=/CRYPTO_POLICY=/g' /etc/sysconfig/sshd Restart sshd: [archy@server ~]$ sudo systemctl restart sshd.service Check with nmap to see if the settings have been applied: [archy@server ~]$ nmap -sV -Pn -p 22 -open -script ssh2-enum-algos 127.0.0.1 Starting Nmap 7.80 ( https://nmap.org ) at 2021-01-26 19:40 CET Nmap scan report for 127.0.0.1 Host is up (0.00096s latency). PORT STATE SERVICE VERSION 22/tcp open ssh OpenSSH 8.0 (protocol 2.0) | ssh2-enum-algos: | kex_algorithms: (1) | curv...

Creating a floating IP between 2 hosts is very handy when dealing with High Availability in dependency of one or more services. I'll use a floating IP between 2 IPA Servers in Order to create an HA-DNS Endpoint so that servers can be upgraded independently. Here's a short description: ipa01.archyslife.lan:   - ip: 172.31.10.5   - service: named-pkcs11   - keepalived-state: MASTER ipa02.archyslife.lan   - ip: 172.31.10.6   - service: named-pkcs11   - keepalived-state: BACKUP First of all, install the 'keepalived' package on all servers: [archy@ipa01 ~]$ sudo yum -y install keepalived [archy@ipa02 ~]$ sudo yum -y install keepalived Now, create the configuration for the main Instance first: [archy@ipa01 ~]$ sudo vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { notification_email { admins@archyslife.lan } notification_email_from root@ipa01.archyslife.lan smtp_server 127.0.0.1 smtp...