Archy's Blog

I'm using plain KVM + Libvirt as my hypervisor of choice in my Homelab since it gives me a lot of flexibility, reliability and performance. Installing VMs using traditional installers allows for customizations during install but if all you're doing is quickly spinning up a VM to test something, pre-built Cloud Images are probably a better choice.  The Cloud Images can be customized though before importing them using tools like virt-sysprep or cloud-init. In this article, I'll be covering a workflow using provided Cloud Images and Cloud Init to bootstrap ephemeral Linux Servers. First, we'll have to download the cloud image, I'll be using a Amazonlinux Cloud Image this time: [root@hyv02 ~]# curl -4 -f -k -L -Z -o '/var/kvm/nfs-vm-templates/amazonlinux-2023-2025-07-21-x86_64.qcow2' -X 'GET' -H 'Accept: application/octet-stream' -H 'User-Agent: curl/1.33.7' https://cdn.amazonlinux.com/al2023/os-images/2023.8.20250721.2/kvm/al2023-kvm-2...

I'm using plain KVM + Libvirt as my hypervisor of choice in my Homelab since it gives me a lot of flexibility, reliability and performance. Installing VMs using traditional installers allows for customizations during install but if all you're doing is quickly spinning up a VM to test something, pre-built Cloud Images are probably a better choice.  The Cloud Images can be customized though before importing them using tools like virt-sysprep or cloud-init . In this Article, I'll be covering my workflow using virt-sysprep with a Alma Cloud Image although any other cloud image should work. [root@hyv02 ~]# curl -4 -f -k -L -Z -o '/var/kvm/nfs-vm-templates/almalinux-9-2025-05-22-x86_64.qcow2' -X 'GET' -H 'Accept: application/octet-stream' https://raw.repo.almalinux.org/almalinux/9/cloud/x86_64/images/AlmaLinux-9-GenericCloud-9.6-20250522.x86_64.qcow2 [root@hyv02 ~]# chown root:root /var/kvm/nfs-vm-templates/almalinux-9-2025-05-22.x86_64.qcow2; chmod 60...

Creating users is typically straightforward as the documentation for the required Ansible modules is comprehensive and easy to navigate. However, working with dictionaries instead of lists can introduce some additional complexity.  For example, let's assume the following dictionary structure is given: usergroups: group1: gid: 10001 name: group1 group2: gid: 10002 name: group2 group3: gid: 10003 name: group3 group4: gid: 10004 name: group4 users: user1: uid: 1985 name: user1 groups: - group1 - group2 sshkeys: - ssh-ed25519 AAAA0 - ssh-ed25519 AAAA1 user2: uid: 1986 name: user2 groups: - group4 - group1 sshkeys: - ecdsa-sha2-nistp384 AAAA0 Ansible provides the dict2items filter which transforms a dictionary into a list of key-value pairs. This transformation allows you to ite...

  When updating Rundeck from version 4.17 to 5.0, the database has to be migrated from v2 to v3. So here's  a quick how-to: First, stop the rundeck service and create a backup: [root@rundeck ~]# systemctl stop rundeckd.service [root@rundeck ~]# mkdir -p /var/backup/rundeck [root@rundeck ~]# tar -cvpf /var/backup/rundeck/rundeck-db-v2-$(date +%F).tar /var/lib/rundeck/data There's a script that can be utilized to migrate the db from v2 to v3 so let's clone that: [root@rundeck ~]# git clone https://github.com/rundeck-plugins/h2-v2-migration.git Now run the migration script against the current database. Ensure that you have a backup so you can rollback if things go haywire: [root@rundeck ~]# cd h2-v2-migration [root@rundeck ~]# sh migration.sh -f /var/lib/rundeck/data/rundeckdb -u 'sa' -p '' -s v2 -d v3 Once that migration is done, the script will create an './output' directory where the db files are stored. You can go ahead and copy th...

Managing mulltiple ssh-key pairs can be made easy by utilizing the power of ssh-agent. However, I'd like to have my ssh-sessions automatically attach to the ssh-agent or start a instance of ssh-agent if it's not already started. Starting the ssh-agent and attaching to it can be achieved by using these commands: [archy@server ~]$ ssh-agent -s > ${HOME}/.ssh/environment-$(hostname -s) [archy@server ~]$ source ${HOME}/.ssh/environment-$(hostname -s) This will create a file named 'environment-server'  in the folder ~/.ssh with all information required to attach it and then source it to attach to the running ssh-agent. Reconnecting to the running ssh-agent can be done using this command again: [archy@server ~]$ source ${HOME}/.ssh/environment-$(hostname -s) Another thing to consider is not starting multiple ssh-agents, so we'll have to check there is a instance of ssh-agent running for the current user and then determine if we should attach to the currently r...

When installing OpenShift or OKD4, all nodes will be using a default chrony config which doesn't necessarily work for all environments, such as firewalled-environments for example. Here's a quick how-to on how to create a custom /etc/chrony.conf for all nodes in your OpenShift Cluster: There are some prerequisites however. Butane is required as well as access to the openshift cluster along with administrative permissions. Start by downloading the 'butane' binary from github: [root@helper01 ~]# BUTANE_VERSION='v0.18.0' [root@helper01 ~]# curl -4kLo '/usr/bin/butane' -X GET "https://github.com/coreos/butane/releases/download/${BUTANEVER}/butane-x86_64-unknown-linux-gnu" [root@helper01 ~]# chown root:root /usr/bin/butane [root@helper01 ~]# chmod 755 /usr/bin/butane Create the two butane configs, first the master nodes: [root@helper01 ~]# cat 99-master-chrony.bu variant: openshift version: 4.13.0 metadata: name: 99-master-chr...

I'm using ansible as a way of remote execution on my foreman / satellite servers. With version 3.5 (foreman) / 6.13 (satellite) I've found that the setting for 'ansible_ssh_private_key_file' is not permanently set after re-running 'foreman-installer'. First, verify that's actually your problem: [root@katello ~]# hammer --no-use-defaults settings info --name 'ansible_ssh_private_key_file' If the value returned is not correct, fix it by using the 'settings set'-subcommand: [root@katello ~]# hammer --no-use-defaults settings set --name 'ansible_ssh_private_key_file' --value '/var/lib/foreman-proxy/ssh/foreman-proxy-ecdsa' Ansible should now work, there's no restart of foreman-proxy required. Feel free to comment and / or suggest a topic.

When updating Rundeck from version 4.0 to 4.1(+), the database has to be migrated from v1 to v2 because the service won't start otherwise. So here's  a quick how-to: First, stop the rundeck service and create a backup: [root@rundeck ~]# systemctl stop rundeckd.service [root@rundeck ~]# mkdir -p /var/backup/rundeck [root@rundeck ~]# tar -cvpf /var/backup/rundeck/rundeck-db-v1-$(date +%F).tar /var/lib/rundeck/data There's a script that can be utilized to migrate the db from v1 to v2 so let's clone that: [root@rundeck ~]# git clone https://github.com/rundeck-plugins/h2-v2-migration.git Now run the migration script against the current database. Ensure that you have a backup so you can rollback if things go haywire: [root@rundeck ~]# cd h2-v2-migration [root@rundeck ~]# /usr/bin/sh migration.sh -f /var/lib/rundeck/data/rundeckdb -u 'sa' -p Once that migration is done, the script will create an './output' directory where the db files are stor...

Depending on what you're configuring, you might have to configure YAML Variables inside of text strings that should not be templated at runtime. There's a solution for templates and for tasks. Let's start with templates, there's '{% raw %}' and '{% endraw %}'. Here's an example: {% raw %} a string that won't be templated and contains a {{ var }} {% endraw %} This will result in the following file content: a string that won't be templated and contains a {{ var }} Now for playbooks, you can use '!unsafe' in front of the line of text. This will make ansible ignore the jinja variable definition and treat it as text.  Now to the problem that spawned this post: creating a mail notification with custom messages in AWX. Here's an example playbook: --- - hosts: localhost become: false delegate_to: localhost collections: - awx.awx tasks: - name: create mail notification awx.a...

When using 'become: true' in one of your tasks and centralized users (389-ds, FreeIPA, Active Directory) the authentication daemon needs to validate if the user is allowed to run sudo on that host. Timeouts can happen for various reasons, so it's best to check /var/log/secure or /var/log/auth for errors. In my case, I got this error: pam_sss(systemd-user:account): Access denied for user executor: 6 (Permission denied) The first thing to check is if the user is allowed to use that service on the host. In my case with FreeIPA, the result is just a hbactest command away: [archy@ipa02 ~]$ ipa hbactest --user 'executor' --host 'logstash02.archyslife.lan' --service 'sudo' | egrep -vi 'not' -------------------- Access granted: True -------------------- Matched rules: allow_executor_all_hosts In further debugging, I found that the swap and ram were heavily utilized on that host. The solution here was to restart the service that was...

If you have a huge amount of hosts, the execution runtimes can really skyrocket. The playbook with the roles took about 90 minutes to complete on ~250 Hosts with no tweaks to the ansible.cfg. First, switch to the directory where your ansible plays reside in and enable the timer callback plugin: [archy@ansible02 /var/ansible]$ echo 'callback_whitelist = timer' >> ansible.cfg This will summarize what tasks took a long time to execute. If there's nothing obvious, you can increase the forks. By default, it's set to '5' in the /etc/ansible/ansible.cfg [archy@ansible02 /var/ansible]$ echo 'forks = 50' >> ansible.cfg Run your playbook again and check the timings. The 'forks' parameter is only limited by cpu and network throughput ... so this might require some tweaking for your specific environment. Using only the 'forks' parameter, the execution runtime of the aforementioned play dropped by -15 minutes. The last performance tw...

You could harden / obfuscate your system by making important files immutable such as the '/etc/sssd/sssd.conf' or '/etc/selinux/config' to prevent automatic changes to these files. I'll template these using ansible to have a deployment workflow so that all my systems are equal and I can make changes in a deployment fashion. Here are some tasks to give you a basic idea: - name: selinux conig immutable block block: - name: configure selinux config template: src: templates/etc/selinux/config.j2 dest: /etc/selinux/config owner: root group: root mode: '0644' tags: - selinux_config - name: set immutable attribute file: path: /etc/selinux/config attr: '+i' rescue: - name: unset immutable attribute file: path: /etc/selinux/config attr:...

Sometimes one specific item in a list needs special treatment or should be used in a different way. In ansible, one method would be to use the 'selectattr()' filter which basically works like a if statement in python looking for a specific attribute value. Here's a code snippet using selectattr to download one specific artifact from a list of artifacts - name: download newest specific artifact loop: "{{ application_artifacts | selectattr('artifact', 'match', application1) | list }}" maven_artifact: group_id: "{{ item.group }}" artifact_id: "{{ item.artifact }}" classifier: "{{ item.classifier }}" extensions: "{{ item.extensions }}" version. "{{ item.version }}" repository_url: "{{ item.repourl }}" dest: /tmp/ validate_certs: false tags: - deploy_artifacts - deploy_application The advantage here would be...

I've been migrating my IAM deployment with FreeIPA to the freeipa.ansible_freeipa collection which worked fine for the most part. I've encountered a false-positive failure when using external groups / external members which will be pulled in using a trust. Here are three posible outcomes: Case 1: The external group is not already a member in which case the ipagroup module will search for, try to add it and return a 'changed' state if it was successful Case 2: The external group is already a member in which case the ipagroup module will return a 'failed' with a message of 'trusted domain object is already a member' which is the false positive Case 3: The external group is not already a member in which case the ipagroup module will search for, try to add it and return a 'failed' state if it was unsuccessful I will focus on case number two for this post. I'm starting with this task in my playbook which behaves exactly as described above: - name...

When using conditionals in ansible, most of the time you'll use 'when' and specify which parameter should be checked. This can however result in a rather lengthy and confusing list of tasks that can be easily reduced. Here's an example of a list of tasks that create users: - name: create regular users | --password, --groups become: true loop: "{{ users }}" user: name: "{{ item.name }}" uid: "{{ item.uid }}" shell: "{{ item.shell | default('/bin/sh') }}" when: not item.groups is defined and not item.password is defined tags: - all - users - name: create regular users | --password, ++groups become: true loop: "{{ users }}" user: name: "{{ item.name }}" uid: "{{ item.uid }}" shell: "{{ item.shell | default('/bin/sh') }}" groups: "{{ item.groups }}" when: ite...

When there's a foreman present in the environment, it's wise to use it as an inventory source for your ansible deployments since it will most likely always be up to date when it comes to hosts (single source of truth).  In order to use ansible as inventory, you'll have to define a *.foreman.yml file in your inventory which will then query the foreman for hosts, groups and variables which can then be used in ansible: [archy@ansible ~/ansible]$ vim inventories/foreman/archyslife.foreman.yml plugin: foreman url: https://katello.archyslife.lan user: roadmin password: !vault | $ANSIBLE_VAULT;1.1;AES256 00000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000...