FreeIPA - Un-bricking DNSSEC after SoftHSM Token Loss

I've been using DNSSEC to sign my internal DNS zones with FreeIPA for quite some time now and have never had any problems. But suddenly, I noticed ipa-dnskeysyncd.service was continuously failing to start, throwing this traceback:

 dnssec-keyfromlabel: fatal: failed to get key RSASHA256: not found   
 ipa-dnskeysyncd.service: Main process exited, code=exited, status=1/FAILURE

Looking deeper into the logs, I could see that LDAP was happily reporting it was adding key metadata to my zones, but the local token was instantly rejecting it:

 The public key was not found at: pkcs11:object=...

The root cause turned out to be a cryptographic "split-brain" situation. The metadata stored in the LDAP directory was correct, but the actual cryptographic private/public key material stored by the local SoftHSM token was missing.

You can verify if your saved keys are actually in the token by pointing pkcs11-tool at the FreeIPA SoftHSM database:

 [root@ipa03 ~]# export SOFTHSM2_CONF=/etc/ipa/dnssec/softhsm2.conf  
 [root@ipa03 ~]# pkcs11-tool --module /usr/lib64/pkcs11/libsofthsm2.so -l -O --pin $(cat /var/lib/ipa/dnssec/softhsm_pin)

The logical first step would be to just disable DNSSEC on the affected zones using the CLI. Don't bother; this won't work. Toggling that flag updates LDAP, but ipa-dnskeysyncd is the daemon responsible for actually applying that change. Because it is trapped in a crash loop during its initial startup scan, it will die before it is ever able to read your change.

The only way forward here is performing open-heart surgery: purging the orphaned keys directly from the LDAP directory. Replication and a restart of the DNS services will take care of the rest.

First, grab an admin ticket, generate a list of all broken DNSSEC keys in LDAP, and stage them for deletion:

 [root@ipa03 ~]# kinit admin  
 [root@ipa03 ~]# ldapsearch -Y GSSAPI -o ldif-wrap=no -b 'cn=dns,dc=archyslife,dc=lan' '(objectclass=idnsSecKey)' dn | awk '/^dn: / {print $2}' > /tmp/orphaned-dnssec-keys.txt

Then, bulk-delete the corrupted keys from LDAP:

 [root@ipa03 ~]# ldapdelete -Y GSSAPI -f /tmp/orphaned-dnssec-keys.txt

With the poison pills removed, restart the sync daemon. It will start cleanly and generate fresh keys:

 [root@ipa03 ~]# systemctl restart ipa-dnskeysyncd.service

Now it's time to clean up OpenDNSSEC's backlog. Because FreeIPA generates new keys, they will be stuck in a "waiting" state. Search for the waiting keys and grab their CKA_ID:

 [root@ipa03 ~]# ods-enforcer key list -v | grep -iIE 'waiting for ds-seen'

Next, force the transition for your zones by telling OpenDNSSEC the DS records have been seen:

 [root@ipa03 ~]# ods-enforcer key ds-seen --zone ${DNS_ZONE} --cka_id ${CKA_ID}

As a final step, ensure the physical key material is properly replicated via Custodia and loaded into memory on all your replicas. The easiest way to do this is to restart the IPA stack on all remaining ipa servers:

 [root@ipa01 ~]# ipactl stop; sleep 10; ipactl start  
 [root@ipa02 ~]# ipactl stop; sleep 10; ipactl start

Finally, verify that your servers are returning the new DNSSEC signatures. Run this against each of your FreeIPA DNS servers to confirm the RRSIG is attached:

 [root@client ~]# dig +dnssec +short @ip.of.dns.server SOA archyslife.lan

Feel free to comment and / or suggest a topic.

Comments

Dynamic DNS with BIND and ISC-DHCP

I personally prefer to work with hostnames instead of ip-addresses. If you have anything like freeipa or active directory, it will do that for you by registering the client you added to your realm to the managed dns and edit the records dynamically. We can achieve the same goal with just bind and isc-dhcp. I'll use a raspberry pi with raspbian 9 for this setup. So here is a quick tutorial on how to configure the isc-dhcp-server to dynamically update bind. First set a static ip to your server. [archy@ddns ~]$ sudo vim /etc/network/interfaces # interfaces(5) file used by ifup(8) and ifdown(8) # Please note that this file is written to be used with dhcpcd # For static IP, consult /etc/dhcpcd.conf and 'man dhcpcd.conf' # Include files from /etc/network/interfaces.d: source-directory /etc/network/interfaces.d auto eth0 iface eth0 inet static address 172.31.30.5 network 172.31.30.0 broadcast 172.31.30.255 netmask 255.255.255.0 ...

Push logs and data into elasticsearch - Part 2 Mikrotik Logs

This is only about the setup of different logging, one being done with Filebeat and the other being done with sending logging to a dedicated port opened in Logstash using the TCP / UDP Inputs. Prerequesites: You'll need a working Elasticsearch Cluster with Logstash and Kibana. Start by getting the Log Data you want to structure parsed correctly. Mikrotik Logs are a bit difficult since they show you Data in the interface which is already enriched with Time / Date. That means a message that the remote logging will send to Logstash will look like this: firewall,info forward: in:lan out:wan, src-mac aa:bb:cc:dd:ee:ff, proto UDP, 172.31.100.154:57061->109.164.113.231:443, len 76 You can check them in the grok debugger and create your own filters and mapping. The following is my example which might not fit your needs. Here are some custom patterns I wrote for my pattern matching: MIKROTIK_DATE \b(?:jan(?:uary)?|feb(?:ruary)?|mar(?:ch)?|apr(?:il)?|may|jun(?:e)?|jul(?...

LACP-Teaming on CentOS 7 / RHEL 7

What is teaming? Teaming or LACP (802.3ad) is a technique used to bond together multiple interfaces to achieve higher combined bandwith. NOTE: every clients speed can only be as high as the single link speed of one of the members. That means, if the interfaces I use in the bond have 1 Gigabit, every client will only have a maximum speed of 1 Gigabit. The advantage of teaming is, that it can handle multiple connections with 1 Gigabit. How many connections depends on the amount of your network cards. I'm using 2 network cards for this team on my server. That means I can handle 2 Gigabit connections at full rate on my server provided the rest of the hardware can deliver that speed. There also exists 'Bonding' in the Linux world. They both do the same in theory but for a detailed comparison check out this article about teaming in RHEL7 . To create a teaming-interface, we will first have to remove all the interface configurations we've done on the (soon to be) sla...

FreeIPA - Integrating your DHCPD dynamic Updates into IPA

I recently went over my network configuration and noticed that the dhcp-leases were not pushed into the IPA-DNS yet. So I thought, why not do it now. The setup is very similar to setting it up on a single bind instance not managed by IPA (I've already written a guide about this here ). My setup is done with the following hosts: ipa01.archyslife.lan - 172.31.0.1 inf01.archyslife.lan - 172.31.0.5 First of all, create a rndc-key: [archy@ipa01 ~]$ sudo rndc-confgen -a -b 512 This will create the following file '/etc/rndc-key' [archy@ipa01 ~]$ sudo cat /etc/rndc.key key "rndc-key" { algorithm hmac-md5; secret "secret_key_here=="; }; We also need to make named aware of the rndc-key and allow our remote dhcp server to write dns entries: [archy@ipa01 ~]$ sudo vim /etc/named.conf ... include "/etc/rndc-key"; controls { inet 172.31.0.1 port 953 allow { 172.31.0.5; } keys ...

Creating a pgpool-II based PostgreSQL Cluster

This time I'm going to do a small and quick walkthrough for a postgresql cluster install. I assume you have a clean install of CentOS 7.3 with all updates. The configuration itself is surprisingly simple. The enviroment I'm working with is: Node1: Hostname: pgsql01.archyslife.lan IP: 172.31.10.31 Member of IPA-Domain Selinux: enforcing Node2: Hostname: pgsql02.archyslife.lan IP: 172.31.10.32 Member of IPA-Domain Selinux: enforcing Cluster: Main Node: pgsql01.archyslife.lan Replica: pgsql02.archyslife.lan Virtual IP: 172.31.10.33 for the sake completeness I'll be adding a A-Record entry in the IPA-DNS. Let's start with the configuration of each node. First I will completely setup the Master without restarting the services, afterwards the replica will follow. Steps necessary for both nodes. Add the pgsql-repo to yum. [archy@pgsql01 ~]$ sudo yum -y install https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64/pgdg-centos96-9.6...

Archy's Blog

Search This Blog