Skip to main content

OKD - Create a Homelab Cluster - Prep and Bootstrap

We'll be working on the Servers that are surrounded by the continous lines in this drawing:

Most of the Setup is already done, from here on out the heavylifting will be done from the installer. But first, there's still a few small things left to do: getting the installation artifacts.

Specifically, I'm talking about these artifacts that still need to be downloaded:

  • fedora coreos image
  • openshift-installer
  • openshift-client
  • helm
  • butane
Since we've set up a shared-storage for the webservers that will host these files, they will only need to be downloaded once and can be served from the interal share. I'll download all the artifacts on one of the helper nodes:
 [archy@helper01 ~]$ sudo -Hiu root  
 [root@helper01 ~]# curl -4kLo '/var/www/html/okd/fcos-39.iso' -X 'GET' -H 'Accept: application/octet-stream' -H 'User-Agent: curl/7.76.1' 'https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/39.20240210.3.0/x86_64/fedora-coreos-39.20240210.3.0-live.x86_64.iso'  
 [root@helper01 ~]# curl -4kLo '/var/www/html/okd4/fcos-39.raw.xz' -X 'GET' -H 'Accept: application/octet-stream' -H 'User-Agent: curl/7.76.1' 'https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/39.20240210.3.0/x86_64/fedora-coreos-39.20240210.3.0-metal.x86_64.raw.xz'  
 [root@helper01 ~]# curl -4kLo '/var/www/html/okd4/fcos-39raw.xz.sig' -X 'GET' -H 'Accept: application/octet-stream' -H 'User-Agent: curl/7.76.1' 'https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/39.20240210.3.0/x86_64/fedora-coreos-39.20240210.3.0-metal.x86_64.raw.xz.sig'  
 [root@helper01 ~]# curl -4kLo '/var/www/html/okd4/openshift-install-linux.tar.gz' -X 'GET' -H 'Accept: application/octet-stream' -H 'User-Agent: curl/7.76.1' 'https://github.com/okd-project/okd/releases/download/4.15.0-0.okd-2024-03-10-010116/openshift-install-linux-4.15.0-0.okd-2024-03-10-010116.tar.gz'  
 [root@helper01 ~]# curl -4kLo '/var/www/html/okd4/openshift-client-linux.tar.gz' -X 'GET' -H 'Accept: application/octet-stream' -H 'User-Agent: curl/7.76.1' 'https://github.com/okd-project/okd/releases/download/4.15.0-0.okd-2024-03-10-010116/openshift-client-linux-4.15.0-0.okd-2024-03-10-010116.tar.gz'  
 [root@helper01 ~]# curl -4kLo '/var/www/html/okd4/helm.tar.gz' -X 'GET' -H 'Accept: application/octet-stream' -H 'User-Agent: curl/7.76.1' 'https://get.helm.sh/helm-v3.14.4-linux-amd64.tar.gz'  
 [root@helper01 ~]# curl -4kLo '/var/www/html/okd4/butane' -X 'GET' -H 'Accept: application/octet-stream' -H 'User-Agent: curl/7.76.1' 'https://github.com/coreos/butane/releases/download/v0.20.0/butane-x86_64-unknown-linux-gnu'  
 [root@helper01 ~]# chown -R apache:apache /var/www/html/okd4  
 [root@helper01 ~]# chmod -R 755 /var/www/html/okd4  
 [root@helper01 ~]# restorecon -rv /var/www/html
All artifacts required for the installation are downloaded. Next step: installing the required binaries and prepare the environment.

The only two required artifacts are openshift-install and openshift-client. I'll also install helm and butane for now since they'll be useful later on when customizing the cluster or installing applications. The installation of all these is just unarchiving and moving the binaries to /usr/bin since they're all golang-based.

NOTE: this step will have to be repeated on all helper nodes
 [root@helper01 ~]# mkdir -p -m 755 /var/tmp/install  
 [root@helper01 ~]# tar -xvpzf /var/www/html/okd4/openshift-install-linux.tar.gz -C /var/tmp/install/  
 [root@helper01 ~]# tar -xvpzf /var/www/html/okd4/openshift-client-linux.tar.gz -C /var/tmp/install/  
 [root@helper01 ~]# tar -xvpzf /var/www/html/okd4/helm.tar..gz -C /var/tmp/install/  
 [root@helper01 ~]# install -o 'root' -g 'root' -m '755' /var/tmp/install/{openshift-install,oc} /usr/bin/  
 [root@helper01 ~]# install -o 'root' -g 'root' -m '755' /var/tmp/install/linux-amd64/helm /usr/bin/  
 [root@helper01 ~]# install -o 'root' -g 'root' -m '755' /var/www/html/okd4/butane /usr/bin/
With all the binaries installed, I'll create a dedicated install directory. This will make it easier to sync files and manifests between nodes.
 [root@helper01 ~]# mkdir -p -m 750 openshift-installer
Additionally, we'll also have to create a ssh-key in order to be able to ssh as the 'core' user in case kubelet fails. This will also be prove usefull when checking the bootstrap process. I'll also start a ssh-agent to attach the key to and make it attachable for future sessions.
 [root@helper01 ~]# ssh-keygen -a 128 -t ed25519 -C "okd $(date +%F)" -f "${HOME}/.ssh/$(date +%F)-okd-ed25519" -N '' -Z 'chacha20-poly1305@openssh.com'  
 [root@helper01 ~]# ssh-agent -s | tee ~/.ssh/environment-$(hostname -s) && source ~/.ssh/environment-$(hostname -s)  
 [root@helper01 ~]# ssh-add ~/.ssh/$(date +%F)-ocp-ed25519  
 [root@helper02 ~]# ssh-keygen -a 128 -t ed25519 -C "okd $(date +%F)" -f "${HOME}/.ssh/$(date +%F)-okd-ed25519" -N '' -Z 'chacha20-poly1305@openssh.com'  
 [root@helper02 ~]# ssh-agent -s | tee ~/.ssh/environment-$(hostname -s) && source ~/.ssh/environment-$(hostname -s)  
 [root@helper02 ~]# ssh-add ~/.ssh/$(date +%F)-ocp-ed25519
I'll write the install-config.yaml on my first helper node and sync it over to the second node later on once all the manifests are generated.
 [root@helper01 ~]# ssh-add -L >> ~/openshift-installer/install-config.yaml
With the last command, the first step is already done to write the install-config.yaml.
We'll also need a pull secret for this which can be obtained from here or paste a a bogus one like '{"auths":{"fake":{"auth": "foo"}}}':
 [root@helper01 ~]# cat pull-secret.txt >> ~/openshift-installer/install-config.yaml
Two blocks of the install-config are already present, now we have to create the rest of it:
 [root@helper01 ~]# vim install-config.yaml
Here's the content of my install-config as an example:
 apiVersion: v1  
 baseDomain: archyslife.lan  
 compute:  
 - hyperthreading: Enabled  
   name: worker  
   replicas: 0  
 controlPlane:  
   hyperthreading: Enabled  
   name: master  
   replicas: 3  
 metadata:  
   name: okd  
 networking:  
   clusterNetwork:  
   - cidr: 10.128.0.0/14  
     hostPrefix: 23  
   networkType: OVNKubernetes  
   serviceNetwork:  
   - 172.30.0.0/16  
 platform:  
   none: {}  
 fips: false  
 pullSecret: '{"auths": ...}' # content from pull secret of fake pull secret  
 sshKey:  
 - ssh-ed25519 ... # public key from helper node 1  
 - ssh-ed25519 ... # public key from helper node 2
If the install-config.yaml is done, back it up just in case.
 [root@helper01 ~]# if [ ! -d /var/backup/openshift-installer ];then mkdir -p -m 700 /var/backup/openshift-installer; fi  
 [root@helper01 ~]# cp -v ~/openshift-installer/install-config.yaml /var/backup/openshift-installer/install-config.yaml
Now, we can continue to create the manifests. I'll also make sure to create a backup after each step just so I have a easy way to get the files and review them if needed:
 [root@helper01 ~]# openshift-install create manifests --dir ~/openshift-installer  
 [root@helper01 ~]# tar -cvpf /var/backup/openshift-installer/openshift-installer-0.tar ~/openshift-installer
Since we've specified above that there are 0 worker nodes, the masters will be marked as schedulable. Since there will be worker nodes, this can be reverted:
 [root@helper01 ~]# sed -i 's/mastersSchedulable: true/mastersSchedulable: False/' openshift-installer/manifests/cluster-scheduler-02-config.yml  
 [root@helper01 ~]# tar -cvpf /var/backup/openshift-installer/openshift-installer-1.tar ~/openshift-installer
Since the masters are now tainted again, we can proceed with the ignition configs used by CoreOS:
 [root@helper01 ~]# openshift-install create ignition-configs --dir ~/openshift-installer  
 [root@helper01 ~]# tar -cvpf /var/backup/openshift-installer/openshift-installer-2.tar ~/openshift-installer
The installation files are prepared now and can be synced to the web directory in order to be available to all nodes:
 [root@helper01 ~]# rsync -vrlptgoDxhP -- ~/openshift-installer/ /var/www/html/okd/  
 [root@helper01 ~]# chown -R apache:apache /var/www/html/okd  
 [root@helper01 ~]# chmod -R 755 /var/www/html/okd  
 [root@helper01 ~]# restorecon -rv /var/www/html/okd
Before continuing, I'd recommend testing the availability of the files. A simple curl will do:
 [root@helper01 ~]# KEEPALIVEDVIP='172.31.10.240'  
 [root@helper01 ~]# curl -4kLX 'GET' -H 'Referer: openshift-installer' -H 'User-Agent: curl/8.2.1' -H 'Accept: application/json' 'http://${KEEPALIVEDVIP}:8080/metadata.json' | jq -C
Here's one of the downsides in using the UPI: the installation commands will have to be entered manually on the nodes. So boot the nodes using the coreos iso downloaded previously and enter these commands according to each node type:

Bootstrap Node:
 [core@coreos-installer ~]$ export KEEPALIVEDVIP='172.31.10.240'  
 [core@coreos-installer ~]$ sudo -E coreos-installer install /dev/vda -a x86_64 -s stable -u http://${KEEPALIVEDVIP}:8080/okd/fcos-39.raw.xz -I http://${KEEPALIVEDVIP}:8080/okd/bootstrap.ign --insecure --insecure-ignition
Master Nodes:
 [core@coreos-installer ~]$ export KEEPALIVEDVIP='172.31.10.240'  
 [core@coreos-installer ~]$ sudo -E coreos-installer install /dev/vda -a x86_64 -s stable -u http://${KEEPALIVEDVIP}:8080/okd/fcos-39.raw.xz -I http://${KEEPALIVEDVIP}:8080/okd/master.ign --insecure --insecure-ignition
Worker Nodes:
 [core@coreos-installer ~]$ export KEEPALIVEDVIP='172.31.10.240'  
 [core@coreos-installer ~]$ sudo -E coreos-installer install /dev/vda -a x86_64 -s stable -u http://${KEEPALIVEDVIP}:8080/okd/fcos-39.raw.xz -I http://${KEEPALIVEDVIP}:8080/okd/worker.ign --insecure --insecure-ignition
Once the install commands have been entered and the Nodes rebooted, monitor the bootstrap process:
 [root@helper01 ~]# openshift-install --dir ~/openshift-installer wait for-bootstrap-complete --log-level=info
The initial Bootstrap might take up to 30 minutes. You can follow the bootstrap Process by ssh-ing into the bootstrap node and run the command 'sudo journalctl -b -f -u bootkube.service -u kubelet.service'.

Since the preparations are all done, now is a great time to sync the files between the nodes.
Here's a nice trick using netcat listeners:
 [root@helper02 ~]# nc -lvnp 1337 | tar -xvpf - -C /  
 [root@helper01 ~]# tar -cvpf - /root/openshift-installer /var/backup/openshift-installer | nc -v helper02.okd.archyslife.lan 1337
Creating a archive is required in this case since the netcat listener will only accept one connection.

Feel free to comment and / or suggest a topic.

Labels:

Comments

Post a Comment

Popular posts from this blog

Dynamic DNS with BIND and ISC-DHCP

I personally prefer to work with hostnames instead of ip-addresses. If you have anything like freeipa or active directory, it will do that for you by registering the client you added to your realm to the managed dns and edit the records dynamically. We can achieve the same goal with just bind and isc-dhcp. I'll use a raspberry pi with raspbian 9 for this setup. So here is a quick tutorial on how to configure the isc-dhcp-server to dynamically update bind. First set a static ip to your server. [archy@ddns ~]$ sudo vim /etc/network/interfaces # interfaces(5) file used by ifup(8) and ifdown(8) # Please note that this file is written to be used with dhcpcd # For static IP, consult /etc/dhcpcd.conf and 'man dhcpcd.conf' # Include files from /etc/network/interfaces.d: source-directory /etc/network/interfaces.d auto eth0 iface eth0 inet static address 172.31.30.5 network 172.31.30.0 broadcast 172.31.30.255 netmask 255.255.255.0 ...
Post a Comment
Read more

LACP-Teaming on CentOS 7 / RHEL 7

What is teaming? Teaming or LACP (802.3ad) is a technique used to bond together multiple interfaces to achieve higher combined bandwith. NOTE: every clients speed can only be as high as the single link speed of one of the members. That means, if the interfaces I use in the bond have 1 Gigabit, every client will only have a maximum speed of 1 Gigabit. The advantage of teaming is, that it can handle multiple connections with 1 Gigabit. How many connections depends on the amount of your network cards. I'm using 2 network cards for this team on my server. That means I can handle 2 Gigabit connections at full rate on my server provided the rest of the hardware can deliver that speed. There also exists 'Bonding' in the Linux world. They both do the same in theory but  for a detailed comparison check out this  article about teaming in RHEL7 . To create a teaming-interface, we will first have to remove all the interface configurations we've done on the (soon to be) sla...
8 comments
Read more

Push logs and data into elasticsearch - Part 2 Mikrotik Logs

This is only about the setup of different logging, one being done with Filebeat and the other being done with sending logging to a dedicated port opened in Logstash using the TCP / UDP Inputs. Prerequesites: You'll need a working Elasticsearch Cluster with Logstash and Kibana. Start by getting the Log Data you want to structure parsed correctly. Mikrotik Logs are a bit difficult since they show you Data in the interface which is already enriched with Time / Date. That means a message that the remote logging will send to Logstash will look like this: firewall,info forward: in:lan out:wan, src-mac aa:bb:cc:dd:ee:ff, proto UDP, 172.31.100.154:57061->109.164.113.231:443, len 76 You can check them in the grok debugger and create your own filters and mapping. The following is my example which might not fit your needs. Here are some custom patterns I wrote for my pattern matching: MIKROTIK_DATE \b(?:jan(?:uary)?|feb(?:ruary)?|mar(?:ch)?|apr(?:il)?|may|jun(?:e)?|jul(?...
9 comments
Read more

FreeIPA - Integrating your DHCPD dynamic Updates into IPA

I recently went over my network configuration and noticed that the dhcp-leases were not pushed into the IPA-DNS yet. So I thought, why not do it now. The setup is very similar to setting it up on a single bind instance not managed by IPA (I've already written a guide about this here ). My setup is done with the following hosts: ipa01.archyslife.lan - 172.31.0.1 inf01.archyslife.lan - 172.31.0.5 First of all, create a rndc-key: [archy@ipa01 ~]$ sudo rndc-confgen -a -b 512 This will create the following file '/etc/rndc-key' [archy@ipa01 ~]$ sudo cat /etc/rndc.key key "rndc-key" { algorithm hmac-md5; secret "secret_key_here=="; }; We also need to make named aware of the rndc-key and allow our remote dhcp server to write dns entries: [archy@ipa01 ~]$ sudo vim /etc/named.conf ... include "/etc/rndc-key"; controls { inet 172.31.0.1 port 953 allow { 172.31.0.5; } keys ...
2 comments
Read more

SSSD - Debugging PAM permission denied

Sometimes there's weird errors in IT that occur on random chance. I've had such an encounter with SSSD in combination with IPA(+AD-Trust) recently, where only sometimes, a connection to one of the IPA-Servers would fail with this error: Jul 13 13:36:42 ipa02.archyslife.lan sshd[3478]: pam_sss(sshd:account): Access denied for user runner: 4 (System error) Jul 13 13:36:42 ipa02.archyslife.lan sshd[3478]: fatal: Access denied for user runner by PAM account configuration [preauth] In my case, it was only happening sometimes when running a basic system setup role using ansible on every host in the entire environment. This way, there was no consistent pattern besides being the same host every time if it failed. First up, add the 'debug_level=X' to every section required in the /etc/sssd/sssd.conf where X is a number from 1 to 10 with 10 being the most verbose. Afterward, restart sssd and check the logs for any obvious problems. 1) If you are using local users, check the...
Post a Comment
Read more