Archy's Blog

Posts

OCP4 - Subpaths with routes

This is going to be a short one since the Documentation is quite good. OpenShift Routes allow for rewriting paths to subfolders. Here's an example using the whoogle applicaiton with a dark and light theme: One option to solve this without going the ingress route is using the built-in route capabilities of rewriting the url. Here are the three route-definitions I used: [ 22:43:10 ] - archy ~/git/openshift/search> cat openshift/dark-route.yml --- apiVersion: route.openshift.io/v1 kind: Route metadata: name: search namespace: search labels: app: search spec: host: search.example.com path: / port: targetPort: 8080 to: kind: Service name: search ... [ 22:43:20 ] - archy ~/git/openshift/search> cat openshift/dark-route.yml --- apiVersion: route.openshift.io/v1 kind: Route metadata: name: search namespace: search labels: app: search-dark annotations:

OCP4 - operator cache requires rebuild

After upgrading my OKD cluster to 4.15.0-0.okd-2024-03-10-010116, I noticed that the certified-operators pod in the namespace 'openshift-marketplace' was constantly crashing. Digging a little deeper, I found this to be the root cause: [archy@helper01 ~]$ oc -n openshift-marketplace logs pod/certified-operators-ckpbk -c registry-server time="2024-03-22T07:54:45Z" level=info msg="starting pprof endpoint" address="localhost:6060" time="2024-03-22T07:54:45Z" level=fatal msg="cache requires rebuild: cache reports digest as \"e1eec9cd4a58db98\", but computed digest is \"8fcd2934467d9214\"" I fixed this by disabling and re-enabling the operator source (certified-operators in my case): [archy@helper01 ~]$ oc edit operatorhubs.config.openshift.io/cluster ... spec: disableAllDefaultSources: true sources: - disabled: false name: community-operators - disabled: true

Podman - Change container image storage location

Depending on the amount of images and the size of each image, the backing storage requirements can grow quite large. I'll reconfigure podman to use a new mountpoint to use as backing storage for container images. First, some disk-configuration is required. I'll be using logical volumes for easy volume management: [root@podman02 ~]# pvcreate /dev/vdb [root@podman02 ~]# vgcreate vg_data /dev/vdb [root@podman02 ~]# lvcreate -n lv_var_podman -L 20G [root@podman02 ~]# mkfs -t xfs /dev/vg_data/lv_var_podman [root@podman02 ~]# cat << EOF >> /etc/fstab /dev/vg_data/lv_var_podman /var/podman xfs defaults 0 0 EOF [root@podman02 ~]# systemctl daemon-reload [root@podman02 ~]# mount -a Now, let's reconfigure the storage setting for a user. I'll be using my user as an example. [root@podman02 ~]# mkdir -p -m 750 /var/podman/archy [root@podman02 ~]# chown -R archy:archy /var/podman/archy [root@podman02 ~]# sudo -Hiu archy [archy@podman02

Oracle Linux - Replace uekr with default rhel kernel

By default, Oracle Enterprise Linux (OEL) will use the 'Unbreakable Enterprise Kernel (UEK)' instead of the usual kernel shipped with rhel. Some workloads however require running the rhel kernel due to compatibility. Replacing the kernel is actually fairly simple but it requires a reboot, so downtime is expected. I'll be demonstrating this using oel8 but the steps should be similar for oel7 or oel9. First, edit the default kernel sysconfig-file: [root@oel8 ~]# sed -e 's/DEFAULTKERNEL=kernel-uek/DEFAULTKERNEL=kernel/g' -i /etc/sysconfig/kernel Next, install the 'kernel-core' package: [root@oel8 ~]# dnf -4y --refresh install kernel-core List all available kernel images using grubby: [root@oel8 ~]# grubby --info ALL | grep -iE '^kernel' Now set the rhel kernel to boot on next reboot: [root@oel8 ~]# grubby --set-default /boot/vmlinuz-4.18.0-513.18.1.el8_9.x86_64 A reboot is now required in order to remove the kernel-uek packages: [root@oel8

Rundeck - Migrate the H2 DB from v2 to v3

When updating Rundeck from version 4.17 to 5.0, the database has to be migrated from v2 to v3. So here's a quick how-to: First, stop the rundeck service and create a backup: [root@rundeck ~]# systemctl stop rundeckd.service [root@rundeck ~]# mkdir -p /var/backup/rundeck [root@rundeck ~]# tar -cvpf /var/backup/rundeck/rundeck-db-v2-$(date +%F).tar /var/lib/rundeck/data There's a script that can be utilized to migrate the db from v2 to v3 so let's clone that: [root@rundeck ~]# git clone https://github.com/rundeck-plugins/h2-v2-migration.git Now run the migration script against the current database. Ensure that you have a backup so you can rollback if things go haywire: [root@rundeck ~]# cd h2-v2-migration [root@rundeck ~]# sh migration.sh -f /var/lib/rundeck/data/rundeckdb -u 'sa' -p '' -s v2 -d v3 Once that migration is done, the script will create an './output' directory where the db files are stored. You can go ahead and copy th

Foreman - freeipa_register will only create rsa keys

So this is a weird one that apparently existed for quite some time now. I've only noticed after the latest Upgrade to Foreman 3.9 and Katello 4.11 that the 'freeipa_register' snippet only creates RSA keys for sshd by default. However, I prefer to have all three key types generated: ed25519 ecdsa rsa However, in order to accomplish that, we'll have to modify the 'freeipa_register' privisioning snippet and add the other keys: Before: <% elsif os_major > 7 %> /usr/libexec/openssh/sshd-keygen rsa <% end -%> After: <% elsif os_major > 7 %> /usr/libexec/openssh/sshd-keygen ed25519 /usr/libexec/openssh/sshd-keygen ecdsa /usr/libexec/openssh/sshd-keygen rsa <% end -%> Now the hosts automatically installed by katello will have all three key-types present in the ipa configuration. There's another article that might be of use for regenerating sshfp records for hosts: FreeIPA - Regenerate sshfp records .

FreeIPA - reset / remove WebUI sessions

Sometimes the WebUI Session hangs or just does weird things so it can be of use to Reset the Web Sessions from time to time, note that they will also be reset on a server reboot. First, stop the ipa services and gssproxy.service: [root@ipa02 ~]# ipactl stop [root@ipa02 ~]# systemctl stop gssproxy.service Then remove the cached Sessions from '/run/ipa/ccaches': [root@ipa02 ~]# rm -f /run/ipa/ccaches/* Now start the ipa services again: [root@ipa02 ~]# ipactl start [root@ipa02 ~]# systemctl start gssproxy.service All WebUI sessions are now reset. Note that the ccache-files have the name of the credential, so it's also possible to just reset Sessions for selected users. Feel free to comment and / or suggest a topic.

Foreman - Upgrade to Foreman 3.9 and Katello 4.11

NOTE: this guide exists for Upgrading from v3.7 to v3.8 as well --> here With the current version, the official documentation is quite good and can be referenced. I would recommend executing all of these commands in a tmux session so that your session will remain on the server in case anything happens to your workstation. Start by checking for running tasks that would prohibit an update: [root@katello01 ~]# foreman-rake katello:upgrade_check Next, update the katello host and reboot if yum tells you to: [root@katello01 ~]# dnf -4y --refresh update [root@katello01 ~]# dnf needs-restarting -r If there were any updates to foreman-related packages, make sure foreman is in a consistent state: [root@katello01 ~]# foreman-maintain service stop [root@katello01 ~]# foreman-installer --scenario katello When the katello services have started again, upgrade the release-rpms: [root@katello01 ~]# dnf -4y update https://yum.theforeman.org/releases/3.9/el8/x86_64/foreman-releas

OCP4 - Manage operator sources

I've recently needed to install the min.io operator and noticed that it's not available using a unmodified version of the OperatorHub Configuration. By default, okd4 only enables the 'community-operators' source which does not contain minio. So here's how to enable it: First, log in to the cluster: [13:03:00 - archy@helper01 ~]$ oc login -u 'archy' -p '' --insecure-skip-tls-verify 'https://api.okd.archyslife.lan:6443' Next, we need to edit the operatorhub's spec.sources list: [13:03:12 - archy@helper01 ~]$ oc edit operatorhubs.config.openshift.io/cluster spec: disableAllDefaultSources: true sources: - disabled: false name: community-operators - disabled: false name: certified-operators After the manifest has been applied, check the available operator sources in the WebUI and pods: [13:05:24 - agerth@helper01 ~]$ oc -n openshift-marketplace get pods -o wide --field-selector status.phase=

ssh - automatically start or attach to ssh-agent

Managing mulltiple ssh-key pairs can be made easy by utilizing the power of ssh-agent. However, I'd like to have my ssh-sessions automatically attach to the ssh-agent or start a instance of ssh-agent if it's not already started. Starting the ssh-agent and attaching to it can be achieved by using these commands: [archy@server ~]$ ssh-agent -s > ${HOME}/.ssh/environment-$(hostname -s) [archy@server ~]$ source ${HOME}/.ssh/environment-$(hostname -s) This will create a file named 'environment-server' in the folder ~/.ssh with all information required to attach it and then source it to attach to the running ssh-agent. Reconnecting to the running ssh-agent can be done using this command again: [archy@server ~]$ source ${HOME}/.ssh/environment-$(hostname -s) Another thing to consider is not starting multiple ssh-agents, so we'll have to check there is a instance of ssh-agent running for the current user and then determine if we should attach to the currently r

Synology - Multiple VLANs on the same interface

Enabling mutliple VLANs from the WebUI appears to not be possible. Luckily since it's linux under the hood, we can just edit the config files and add multiple vlans and restart the network service. First, edit the current config for the network interface, in my case 'eth0': root@synology:~# cat << EOF > /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 BOOTPROTO=none ONBOOT=yes EOF Now, create the config files for each vlan. I'll create them in a simple for-loop using bash: root@synology:~# for vlan in 10 11 12 13; do cat << EOF > /etc/sysconfig/network-scripts/ifcfg-eth0.${vlan} DEVICE=eth0.${vlan} VLAN_ROW_DEVICE=eth0 VLAN_ID=${vlan} ONBOOT=yes BOOTPROTO=static IPADDR=172.31.${vlan}.248 NETMASK=255.255.255.0 IPV6INIT=off EOF done After all the VLAN-Definitions have been created, make sure to restart the network service: root@synology:~# /etc/rc.network restart Once the Service is restarted, the vlans should be present:

Foreman - Upgrade to Foreman 3.8 and Katello 4.10

NOTE: this guide exists for Upgrading from v3.6 to v3.7 as well --> here With the current version, the official documentation is quite good and can be referenced. I would recommend executing all of these commands in a tmux session so that your session will remain on the server in case anything happens to your workstation. Start by checking for running tasks that would prohibit an update: [root@katello01 ~]# foreman-rake katello:upgrade_check Next, update the katello host and reboot if yum tells you to: [root@katello01 ~]# dnf -4y --refresh update [root@katello01 ~]# dnf needs-restarting -r If there were any updates to foreman-related packages, make sure foreman is in a consistent state: [root@katello01 ~]# foreman-maintain service stop [root@katello01 ~]# foreman-installer --scenario katello When the katello services have started again, upgrade the release-rpms: [root@katello01 ~]# dnf -4y update https://yum.theforeman.org/releases/3.8/el8/x86_64/foreman-release

Foreman - Configure dhcp-proxy using the remote_isc plugin

I have run my foreman-server up until recently without the dhcp-capsule feature enabled but I decided to change that to have proper integration so here's a quick how-to: First, generate the omapi_key and get the hash: [root@dhcp ~]# dnssec-keygen -r /dev/urandom -a HMAC-MD5 -b 512 -n HOST omapi_key [root@dhcp ~]# grep -iE '^Key' Komapi_key*.private | cut -d ' ' -f 2- Now, prepare the dhcp-server config by appending these options: [root@dhcp ~]# vim /etc/dhcp/dhcpd.conf omapi-port 7911; key omapi_key { algorithm HMAC-MD5; secret "hash-from-command-gathered-earlier"; }; omapi-key omapi_key; Change the permissions for the dhcp folder and config file: [root@dhcp ~]# chmod o+rx /etc/dhcp/ [root@dhcp ~]# chmod o+r /etc/dhcp/dhcpd.conf According to Red Hat's Documentation, the bindmounts are necessary: [root@dhcp ~]# vim /etc/fstab /var/lib/dhcpd /exports/var/lib/dhcpd none bind,auto 0 0 /etc/dhcp /exports/etc/dhcp