Archy's Blog

Sometimes it turns out, your initial sizing of a VM might have been too small. While RAM and CPU can easily be extended, Disk, on the other hand, might be a different story but usually, the process is rather simple. This is how I tend to do extend a disk that is used by LVM. First, resize the VM's disk on the hypervisor, note that this command is specific for KVM [archy@hyv01 ~]$ sudo virsh blockresize --domain repo.archyslife.lan --path /srv/kvm/vm-images/repo.archyslife.lan-disk1.qcow2 --size 250G You can verify your changes by running this command [root@hyv01 ~]# sudo virsh domblkinfo --domain repo.archyslife.lan --device vda --human Capacity: 250.000 GiB Allocation: 199.435 GiB Physical: 250.000 GiB If you are using virtio (like me in this example), the disk should automatically be resized in the VM and you can continue to resize the partition or physical volume. If you're resizing a SATA / SAS / SCSI Disk, you'll have to make the VM resca...

The upgrade between minor versions is fairly simple with Zimbra since the install.sh script covers everything with its interactive installer. First, download the archive used for the installation and unpack it. [archy@zimbra ~]$ curl -O https://files.zimbra.com/downloads/8.8.15_GA/zcs-8.8.15_GA_3869.RHEL7_64.20190918004220.tgz [archy@zimbra ~]$ tar -xvzf zcs-8.8.15_GA_3869.RHEL7_64.20190918004220.tgz I'd recommend moving the installation to tmux (or screen) so that in case of a disconnect, the installation will not break anything and you could just detach / attach to the session. You can create a new tmux session by running [archy@zimbra ~]$ tmux new -s zimbra-upgrade Now switch into the directory where the packages have been unpacked to and run the interactive install.sh [archy@zimbra ~]$ cd zcs-8.8.15_GA_3869.RHEL7_64.20190918004220 [archy@zimbra ~]$ sudo ./install.sh ... Use Zimbra's packaging server? [Y]: y ... Do you with to upgrade? [Y]...

Under certain circumstances, it can happen that foreman's components get out of sync and hosts cannot be registered / unregistered anymore. I've had this problem recently with a pulp upgrade in foreman 1.24.2 and pulp packages: pulp-deb-plugins-1.10.1-1.el7.noarch pulp-docker-plugins-3.2.6-1.el7.noarch pulp-puppet-plugins-2.21.1-1.el7.noarch pulp-puppet-tools-2.21.1-1.el7.noarch pulp-rpm-plugins-2.21.1-1.el7.noarch pulp-selinux-2.21.1-1.el7.noarch pulp-server-2.21.1-1.el7.noarch python-pulp-common-2.21.1-1.el7.noarch python-pulp-deb-common-1.10.1-1.el7.noarch python-pulp-docker-common-3.2.6-1.el7.noarch python-pulp-oid_validation-2.21.1-1.el7.noarch python-pulp-puppet-common-2.21.1-1.el7.noarch python-pulp-repoauth-2.21.1-1.el7.noarch python-pulp-rpm-common-2.21.1-1.el7.noarch python-pulp-streamer-2.21.1-1.el7.noarch The result was an 'Error 500' when trying to unregister a host. In order to fix this, you'll need to rerun the installer for f...

So if you are saving data in elastic search and this data contains IP Addresses, you might want to consider enriching your data with GeoIP. I'll take my router logging data (see part 2) and enrich them with GeoIP data. First of all, let's adjust the Index Template and add the GeoIP Longitude, Latitude and Location fields. I'll push the index template using the dev tools in Kibana. PUT _template/mikrotik-log { "index_patterns" : [ "mikrotik-log-*" ], "settings" : { "index" : { "codec" : "best_compression", "refresh_interval" : "5s", "number_of_shards" : "1", "number_of_replicas" : "1" } }, "mappings" : { "numeric_detection" : true, "dynamic_templates" : [ { "string_fields" : { "mapping" :...

I've had to deal with a little bit of automation for foreman / satellite in which we created hosts and also set them up for building them (build: true). Everything works for new hosts but there's a small problem for already existing hosts. With that 'build: true' a new build token will be generated and at the next reboot, hosts would be kickstarted using pxe. The solution is simple, get the changed state of every item in the loop and if 'changed' == true, then build the host. Turns out it was a bit more complicated than I expected since the state of the task will be changed if one item in it is different. Please note that I will be defining all variables in environment-specific inventories. So here's my solution to conditionally build hosts depending if their state in managing them was changed or not. Also, check the indentation before running this. --- - name: Manage foreman hosts hosts: foreman gather_facts: false tasks: ...

This is a common challenge with foreman and I found that a monthly schedule would fit my needs. The promotion will have to go through different environments such as lab + nightly, development, test, production + management. I've added a schema of how this should look like down below. Now keep in mind, that this is the solution that best fits my needs and might not cover all your requirements. However, here's the short script I've created which will be executed by cron daily: #!/usr/bin/env bash ORGANIZATION_LABEL='YOUR_ORGANIZATION' LIFECYCLE_ENVIRONMENT='' DATE_DAY=$(date +%d) WEEK_DAY=$(date +%u) if [[ $DATE_DAY == '01' ]]; then LIFECYCLE_ENVIRONMENT='nightly' for content_view_id in $(hammer content-view list --organization-label $ORGANIZATION_LABEL | egrep -vi 'default' | awk '{print $1}' | egrep '^[1-9]' | egrep -v '14' | awk NF); do hammer content-view publish --id $cont...

This is fairly simple to happen to your daily workflow. If you reboot a server or disconnect your laptop from the current network (say ethernet and switch to wireless) it may occur that your ssh session is not reconnecting automatically and appears to be 'stuck'. The proper escape sequence to disconnect the client would be '~.' Here's an example: [archy@castle-bravo ~]$ ssh archy@example.archyslife.lan Last login: Sat Dec 21 18:40:04 2019 from castle-bravo.archyslife.lan [archy@example ~]$ reboot well, it's stuck ... time to disconnect Connection to example.archyslife.lan closed. [archy@castle-bravo ~]$ What happened here is that by rebooting the server the ssh-server did not have time to appropriately disconnect all sessions and so the client appears to be 'stuck'. In order to disconnect the client, I sent the escape sequence '~.' and drop to my local terminal again. Feel free to comment and / or suggest a topic....

Fedora 31 has been released at the end of October 2019 so it's time for me to update. Fedora Updates have proven to be reliable but Updates can go wrong and for that case, I want to have a fallback that I can utilize to restore my system without much of a hassle. I use LVM for all my Storage so the solution for me was fairly easy: LVM Snapshots. I'm using one volume group for the system named 'vg_base'. If you are copy-pasting the commands, this is the one parameter you'll have to adjust. Without further ado, let's get through the upgrade scenario. While these commands can be executed as user with sudo privileges, I find it easier to do these as root. Install the system-upgrade plugin [root@castle-bravo ~]# dnf -y install dnf-plugin-system-upgrade Now create the snapshots of all the system lvs. Note that you would only need to snapshot the rpmdb as well as the logical volumes containing /etc/, /bin, /sbin, /usr/bin and /usr/sbin but to be on th...

Making consistent Backups is key to successful disaster recovery. I've found LVM and its snapshot functionality very helpful in that regard. So I'll cover how to create a snapshot and back it up in a consistent state and restore reliably. First things first, you'll need to have some free space in your volume group, I'll be demonstrating this in a virtual machine. First up, let's start by creating a snapshot of a volume: [archy@server ~]$ sudo lvmcreate -s -n lv_home_$(date +%Y%m%d) -L 10G /dev/vg_base/lv_home This will create a snapshot with 10G COW Space which will fill up once you are writing to the lvm you've specified above. In my case, it's the '/home' mount point on the system and I'm assuming that while the backup is running, I won't write 10G of data. Keep in mind that it is possible to create automatic extending snapshots using the lvm.conf but I will not cover that here. With the snapshot created we can now mount it ...

Last week I had quite a lot of 'fun' debugging Kerberos mostly in an 'AD Trust - RedHat IDM' Environment. The problem was that user logins were denied even though kinit and id requests worked as expected and showed the correct results. So I ended up troubleshooting the usual suspects, DNS and time (NTP) and there it was ... the timezone was not set up correct on the Windows AD side. Problem number two, kerberized NFS. Kerberized NFS to a client after daylight savings time changed stopped working. I currently live in a country where daylight savings time is a thing. After daylight savings time changed (set back 1 hour), one client was not able to mount the NFS share with an "access denied by server" message. So naturally, I check the usual suspects, DNS and NTP and everything seems to work fine. The Time was correct and DNS SRV records were present. The error was still the time though and here's why: Checking the client with ' timedatectl '...

First of all, what is ansible? Ansible is a simple-to-learn and very powerful configuration management tool. Ansible is not using a client deployed on clients like puppet or salt do, but instead it utilizes SSH Connections to run tasks on the destination hosts. What I'm going to describe here is how to install ansible on a CentOS 7 Host, create the directory structure, write a role and reference to it Alright now, let's get to install ansible. I'll use a CentOS 7 Host for this: [archy@ansible ~]$ sudo yum -y install ansible This will install ansible with all dependencies on your system. So far so good, let's create the directory structure where you'll most of the time be working in and the first host inventory which will be in /srv/ansible/inventories/production/hosts. [archy@ansible ~]$ sudo mkdir -p /srv/ansible/playbooks/{roles,inventories} [archy@ansible ~]$ sudo mkdir /srv/ansible/playbooks/inventories/production [archy@ansible ~]$ sudo vim ...

If you are setting up a Foreman / Red Hat Satellite server in an enterprise environment, you're most likely facing a problem with access to the internet only being allowed using an HTTP Proxy.  Normally, this can be fixed by setting the 'http_proxy' and 'https_proxy' environment variables in your .bashrc / .zshrc, /etc/environment or just by running these commands: Without authentication: [archy@katello ~]$ export http_proxy='http://proxy.archyslife.lan:3128' [archy@katello ~]$ export https_proxy='http://proxy.archyslife.lan:3128' With authentication: [archy@katello ~]$ export http_proxy='http://user:password@proxy.archyslife.lan:3128' [archy@katello ~]$ export https_proxy='http://user:password@proxy.archyslife.lan:3128' Now, the syncing with pulp won't utilize these variables, neither the http_proxy setting that can be done in the foreman GUI. What I've found to be the most useful is editing the katello....

I've written about how to set up a FreeIPA instance and replica for managing your sudo-rules, hbac-rules, users, groups and DNS Zones. Since hardware (and I recommend using mostly hardware for your deployment) ages, you'll have to replace them once in a while and sometimes the replica join might not go as smooth as expected. I'll cover how to fix two problems that occurred to me after I decommissioned my original master Server. First problem, dnarange was not set on the new replica after decommissioning the original master The dnarange is basically your user- and group-id range. If there's no range there and you're trying to create a user, your command will fail with an error saying that there's no dnarange available. In order to fix this, run the following commands: Get the user-id of your admin user which always uses the first available uid in the dnarange when being installed: [archy@ipa03 ~]$ ipa user-show admin | egrep -i 'uid|gid' ...

I've recently made a PoC at work where I build a scalable DNS Infrastructure. Please note that while the schematic includes the Windows Network, I will not go into detail with that one since this is out of scope. Linux Datacenter (internal zones only): FreeIPA / Red Hat IDM is a full identity management solution by Red Hat which integrates the following components: - 389-DS - Krb5kdc - Bind Nameserver - Dogtag CA - Certmonger This is a nice feature pack but I will only focus on the DNS part for now. The bind nameserver will handle internal domains and external forwarding in this setup. Authoritative DNS (external zones only): Knot-DNS is a high-performance authoritative DNS Server. That means, it just covers the zones itself knows about and is authoritative for. It will not forward DNS Queries to its upstream hosts. Now, let's jump into what needs to be done to make this setup reality. I'll start by setting up the Knot-Servers. Start by setting up th...

If for some reason your ds.keytab has been corrupted, for example through time drift in the hardware clocks of your multi-master infrastructure, you'll find yourself with a non-working or very slow krb5kdc. However, this can be fixed fairly fast but you'll have to check your replicas for errors and maybe even replicate the whole infrastructure from a known good replica. Let's get to fixing the corrupted ds.keytab first. All these steps will be done with your authentication services offline, so it's probably the safest to do all of the steps as root. Start by stopping the ipa services on the host: [archy@ipa02 ~]$ sudo su - [root@ipa02 ~]# ipactl stop Next up move the not-working keytab: [root@ipa02 ~]# mv /etc/dirsrv/ds.keytab /etc/dirsrv/ds.keytab-$(date +%Y-%m-%d) In order to fix the keytab, a few services are required to run: [root@ipa02 ~]# start-dirsrv [root@ipa02 ~]# systemctl start krb5kdc.service Next login to the krb5kdc and export...

This is only about the setup of different logging, one being done with Filebeat and the other being done with sending logging to a dedicated port opened in Logstash using the TCP / UDP Inputs. Prerequesites: You'll need a working Elasticsearch Cluster with Logstash and Kibana. Start by getting the Log Data you want to structure parsed correctly. Mikrotik Logs are a bit difficult since they show you Data in the interface which is already enriched with Time / Date. That means a message that the remote logging will send to Logstash will look like this: firewall,info forward: in:lan out:wan, src-mac aa:bb:cc:dd:ee:ff, proto UDP, 172.31.100.154:57061->109.164.113.231:443, len 76 You can check them in the grok debugger and create your own filters and mapping. The following is my example which might not fit your needs. Here are some custom patterns I wrote for my pattern matching: MIKROTIK_DATE \b(?:jan(?:uary)?|feb(?:ruary)?|mar(?:ch)?|apr(?:il)?|may|jun(?:e)?|jul(?...

This is only about the setup of different logging, one being done with Filebeat and the other being done with sending logging to a dedicated port opened in Logstash using the TCP / UDP Inputs. Prerequesites: You'll need a working Elasticsearch Cluster with Logstash and Kibana and an installation of Filebeat on the Host(s) where you get your nginx logs from. Start by getting the Log Data you want to structure parsed correctly. The nginx logs are pretty straight forward, so after checking them out in the grok debugger, I'll have the following structure mapped: %{IP:ClientIP} - %{DATA:username} \[%{NGINXTIMESTAMP:timestamp}%{GREEDYDATA}\] \"%{WORD:method} %{DATA:request_uri} %{DATA:http-version}\" %{RETURNCODE:http_return_code} %{GREEDYDATA} \"%{DATA:server_name}\" \"%{GREEDYDATA}\" Also, I've written some custom patterns: NGINXTIMESTAMP (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])\/\b(?:[Jj]an(?:uary|uar)?|[Ff]eb(?:ruary|rua...