Archy's Blog

By default, Oracle Enterprise Linux (OEL) will use the 'Unbreakable Enterprise Kernel (UEK)' instead of the usual kernel shipped with rhel. Some workloads however require running the rhel kernel due to compatibility. Replacing the kernel is actually fairly simple but it requires a reboot, so downtime is expected. I'll be demonstrating this using oel8 but the steps should be similar for oel7 or oel9. First, edit the default kernel sysconfig-file: [root@oel8 ~]# sed -e 's/DEFAULTKERNEL=kernel-uek/DEFAULTKERNEL=kernel/g' -i /etc/sysconfig/kernel Next, install the 'kernel-core' package: [root@oel8 ~]# dnf -4y --refresh install kernel-core List all available kernel images using grubby: [root@oel8 ~]# grubby --info ALL | grep -iE '^kernel' Now set the rhel kernel to boot on next reboot: [root@oel8 ~]# grubby --set-default /boot/vmlinuz-4.18.0-513.18.1.el8_9.x86_64 A reboot is now required in order to remove the kernel-uek packages: [root@oel8 ...

  When updating Rundeck from version 4.17 to 5.0, the database has to be migrated from v2 to v3. So here's  a quick how-to: First, stop the rundeck service and create a backup: [root@rundeck ~]# systemctl stop rundeckd.service [root@rundeck ~]# mkdir -p /var/backup/rundeck [root@rundeck ~]# tar -cvpf /var/backup/rundeck/rundeck-db-v2-$(date +%F).tar /var/lib/rundeck/data There's a script that can be utilized to migrate the db from v2 to v3 so let's clone that: [root@rundeck ~]# git clone https://github.com/rundeck-plugins/h2-v2-migration.git Now run the migration script against the current database. Ensure that you have a backup so you can rollback if things go haywire: [root@rundeck ~]# cd h2-v2-migration [root@rundeck ~]# sh migration.sh -f /var/lib/rundeck/data/rundeckdb -u 'sa' -p '' -s v2 -d v3 Once that migration is done, the script will create an './output' directory where the db files are stored. You can go ahead and copy th...

So this is a weird one that apparently existed for quite some time now. I've only noticed after the latest Upgrade to Foreman 3.9 and Katello 4.11 that the 'freeipa_register' snippet only creates RSA keys for sshd by default. However, I prefer to have all three key types generated: ed25519 ecdsa rsa However, in order to accomplish that, we'll have to modify the 'freeipa_register' privisioning snippet and add the other keys: Before: <% elsif os_major > 7 %> /usr/libexec/openssh/sshd-keygen rsa <% end -%> After: <% elsif os_major > 7 %> /usr/libexec/openssh/sshd-keygen ed25519 /usr/libexec/openssh/sshd-keygen ecdsa /usr/libexec/openssh/sshd-keygen rsa <% end -%> Now the hosts automatically installed by katello will have all three key-types present in the ipa configuration. There's another article that might be of use for regenerating sshfp records for hosts:  FreeIPA - Regenerate sshfp records ....

Sometimes the WebUI Session hangs or just does weird things so it can be of use to Reset the Web Sessions from time to time, note that they will also be reset on a server reboot. First, stop the ipa services and gssproxy.service: [root@ipa02 ~]# ipactl stop [root@ipa02 ~]# systemctl stop gssproxy.service Then remove the cached Sessions from '/run/ipa/ccaches': [root@ipa02 ~]# rm -f /run/ipa/ccaches/* Now start the ipa services again: [root@ipa02 ~]# ipactl start [root@ipa02 ~]# systemctl start gssproxy.service All WebUI sessions are now reset. Note that the ccache-files have the name of the credential, so it's also possible to just reset Sessions for selected users. Feel free to comment and / or suggest a topic.

   NOTE: this guide exists for Upgrading from v3.7 to v3.8 as well -->  here With the current version, the  official documentation  is quite good and can be referenced. I would recommend executing all of these commands in a tmux session so that your session will remain on the server in case anything happens to your workstation. Start by checking for running tasks that would prohibit an update: [root@katello01 ~]# foreman-rake katello:upgrade_check Next, update the katello host and reboot if yum tells you to: [root@katello01 ~]# dnf -4y --refresh update [root@katello01 ~]# dnf needs-restarting -r If there were any updates to foreman-related packages, make sure foreman is in a consistent state: [root@katello01 ~]# foreman-maintain service stop [root@katello01 ~]# foreman-installer --scenario katello When the katello services have started again, upgrade the release-rpms: [root@katello01 ~]# dnf -4y update https://yum.theforeman.org/releases/3.9/e...

I've recently needed to install the min.io operator  and noticed that it's not available using a unmodified version of the OperatorHub Configuration. By default, okd4 only enables the 'community-operators' source which does not contain minio. So here's how to enable it: First, log in to the cluster: [13:03:00 - archy@helper01 ~]$ oc login -u 'archy' -p '' --insecure-skip-tls-verify 'https://api.okd.archyslife.lan:6443' Next, we need to edit the operatorhub's spec.sources list: [13:03:12 - archy@helper01 ~]$ oc edit operatorhubs.config.openshift.io/cluster spec: disableAllDefaultSources: true sources: - disabled: false name: community-operators - disabled: false name: certified-operators After the manifest has been applied, check the available operator sources in the WebUI and pods: [13:05:24 - agerth@helper01 ~]$ oc -n openshift-marketplace get pods -o wide --field-selector status.phase=...

Managing mulltiple ssh-key pairs can be made easy by utilizing the power of ssh-agent. However, I'd like to have my ssh-sessions automatically attach to the ssh-agent or start a instance of ssh-agent if it's not already started. Starting the ssh-agent and attaching to it can be achieved by using these commands: [archy@server ~]$ ssh-agent -s > ${HOME}/.ssh/environment-$(hostname -s) [archy@server ~]$ source ${HOME}/.ssh/environment-$(hostname -s) This will create a file named 'environment-server'  in the folder ~/.ssh with all information required to attach it and then source it to attach to the running ssh-agent. Reconnecting to the running ssh-agent can be done using this command again: [archy@server ~]$ source ${HOME}/.ssh/environment-$(hostname -s) Another thing to consider is not starting multiple ssh-agents, so we'll have to check there is a instance of ssh-agent running for the current user and then determine if we should attach to the currently r...

Enabling mutliple VLANs from the WebUI appears to not be possible. Luckily since it's linux under the hood, we can just edit the config files and add multiple vlans and restart the network service. First, edit the current config for the network interface, in my case 'eth0': root@synology:~# cat << EOF > /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 BOOTPROTO=none ONBOOT=yes EOF Now, create the config files for each vlan. I'll create them in a simple for-loop using bash: root@synology:~# for vlan in 10 11 12 13; do cat << EOF > /etc/sysconfig/network-scripts/ifcfg-eth0.${vlan} DEVICE=eth0.${vlan} VLAN_ROW_DEVICE=eth0 VLAN_ID=${vlan} ONBOOT=yes BOOTPROTO=static IPADDR=172.31.${vlan}.248 NETMASK=255.255.255.0 IPV6INIT=off EOF done After all the VLAN-Definitions have been created, make sure to restart the network service: root@synology:~# /etc/rc.network restart Once the Service is restarted, the vlans should be present: ...

  NOTE: this guide exists for Upgrading from v3.6 to v3.7 as well -->  here With the current version, the  official documentation  is quite good and can be referenced. I would recommend executing all of these commands in a tmux session so that your session will remain on the server in case anything happens to your workstation. Start by checking for running tasks that would prohibit an update: [root@katello01 ~]# foreman-rake katello:upgrade_check Next, update the katello host and reboot if yum tells you to: [root@katello01 ~]# dnf -4y --refresh update [root@katello01 ~]# dnf needs-restarting -r If there were any updates to foreman-related packages, make sure foreman is in a consistent state: [root@katello01 ~]# foreman-maintain service stop [root@katello01 ~]# foreman-installer --scenario katello When the katello services have started again, upgrade the release-rpms: [root@katello01 ~]# dnf -4y update https://yum.theforeman.org/releases/3.8/el8/x86...

I have run my foreman-server up until recently without the dhcp-capsule feature enabled but I decided to change that to have proper integration so here's a quick how-to: First, generate the omapi_key and get the hash: [root@dhcp ~]# dnssec-keygen -r /dev/urandom -a HMAC-MD5 -b 512 -n HOST omapi_key [root@dhcp ~]# grep -iE '^Key' Komapi_key*.private | cut -d ' ' -f 2- Now, prepare the dhcp-server config by appending these options: [root@dhcp ~]# vim /etc/dhcp/dhcpd.conf omapi-port 7911; key omapi_key { algorithm HMAC-MD5; secret "hash-from-command-gathered-earlier"; }; omapi-key omapi_key; Change the permissions for the dhcp folder and config file: [root@dhcp ~]# chmod o+rx /etc/dhcp/ [root@dhcp ~]# chmod o+r /etc/dhcp/dhcpd.conf According to Red Hat's Documentation, the bindmounts are necessary: [root@dhcp ~]# vim /etc/fstab /var/lib/dhcpd /exports/var/lib/dhcpd none bind,auto 0 0 /etc/dhcp /exports/etc/dhcp ...

Upgrading AWX is made very simple when using the operator since it just requires changing the version of the deployment. I'll be using a awx-deployment in okd-4.13 using the awx operator installed using these yaml files: $ cat 0-awx-namespace.yml --- apiVersion: v1 kind: Namespace metadata: name: awx ... $ cat 1-awx-operatorgroup.yml --- apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: awx-operatorgroup namespace: awx spec: upgradeStrategy: Default targetNamespaces: - awx ... $ cat 2-awx-subscription.yml --- apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: awx-operator namespace: awx spec: channel: alpha installPlanApproval: Automatic name: awx-operator source: community-operators sourceNamespace: openshift-marketplace startingCSV: awx-operator.v2.5.3 ... $ cat 4-awx-deployment.yml --- apiVersion: awx.ansib...

There is this strange error where sssd allows logins using gssapi but does not permit sudo and instead writes the dreaded '4 (System error)' to /var/log/secure: Sep 7 10:48:30 server sudo[676183]: pam_sss(sudo:auth): authentication failure; logname=archy uid=103600028 euid=0 tty=/dev/pts/0 ruser=archy rhost= user=archy Sep 7 10:48:30 server sudo[676183]: pam_sss(sudo:auth): received for user archy: 4 (System error) In my case, it was old kerberos tickets that were not deleted. So The fix should be really simple: [archy@server ~]$ kdestroy -A [archy@server ~]$ kinit Troubleshooting steps that might be of interest might be the date / time, hbac rules and the sssd-cache. [root@server ~]# sudo sssctl cache-expire -E [root@server ~]# sudo systemctl restart sssd.service Feel free to comment and / or suggest a topic.

With current ansible versions, the following modules will result in a error if the destination is on a cifs share using systems that have selinux enabled ansible.builtin.copy ansible.builtin.template The error in question: invalid selinux context: [Errno 95] Operation not supported A possible workaround could be to first create the file using 'ansible.builtin.file' with 'state=touch' which would allow the file to be changed later on. Example: - name: ensure empty files are present with:_items: "{{ files_to_copy + files_to_template }}" ansible.builtin.file: path: "{{ item['dest'] }}" state: touch Depending on the structure of your playbooks, this task could go into the 'pre_tasks' section of your playbook, example for a full playbook: --- - name: test hosts: test user: root gather_facts: false collections: - ansible.posix - community.general vars: src_dir:...

When installing OpenShift or OKD4, all nodes will be using a default chrony config which doesn't necessarily work for all environments, such as firewalled-environments for example. Here's a quick how-to on how to create a custom /etc/chrony.conf for all nodes in your OpenShift Cluster: There are some prerequisites however. Butane is required as well as access to the openshift cluster along with administrative permissions. Start by downloading the 'butane' binary from github: [root@helper01 ~]# BUTANE_VERSION='v0.18.0' [root@helper01 ~]# curl -4kLo '/usr/bin/butane' -X GET "https://github.com/coreos/butane/releases/download/${BUTANEVER}/butane-x86_64-unknown-linux-gnu" [root@helper01 ~]# chown root:root /usr/bin/butane [root@helper01 ~]# chmod 755 /usr/bin/butane Create the two butane configs, first the master nodes: [root@helper01 ~]# cat 99-master-chrony.bu variant: openshift version: 4.13.0 metadata: name: 99-master-chr...