FreeIPA - Recover a broken replica

It happened to me that a node broke in a way that was not feasible to 'repair'. By broken, I mean that I was not able to recover by re-creating all keytabs for the dirserver and named that were invalid according to the logs.

Anyway, here's the way I took to recover the node and make all nodes replicate again

First, log in to a working node and remove the faulty node from the replication:

 [archy@ipa02 ~]$ ipa server-del ipa01.archyslife.lan \
  --ignore-topology-disconnect  
 -----------------------------------------  
 Deleted IPA server "ipa01.archyslife.lan"  
 -----------------------------------------

If you're running on the latest version of 4.9.6 and EL8.5, you might encounter this error.

After you've removed the host, check if there are any traces left in your ldap tree:

 [archy@ipa02 ~]$ ipa server-find  
 ---------------------  
 2 IPA servers matched  
 ---------------------  
  Server name: ipa02.archyslife.lan  
  Min domain level: 1  
  Max domain level: 1  
  Server name: ipa03.archyslife.lan  
  Min domain level: 1  
  Max domain level: 1  
 ----------------------------  
 Number of entries returned 2  
 ----------------------------  
 [archy@ipa02 ~]$ ipa-replica-manage list  
 ipa02.archyslife.lan: master  
 ipa03.archyslife.lan: master

Check for any leftover replica update vectors:

 [archy@ipa02 ~]$ sudo ipa-replica-manage list-ruv  
 [sudo] password for archy:   
 Directory Manager password:   
 Replica Update Vectors:  
     ipa02.archyslife.lan:389: 11  
     ipa03.archyslife.lan:389: 13  
 Certificate Server Replica Update Vectors:  
     ipa02.archyslife.lan:389: 12  
     ipa03.archyslife.lan:389: 14

If there are any replica update vectors left over, you can use 'ipa-replica-manage clean-ruv ID' to remove them. Once everything is clear, you can continue on uninstalling the ipa-server on the faulty replica and reboot it just to be sure.

 [root@ipa01 ~]# ipa-server-install --uninstall  
 [root@ipa01 ~]# reboot

Once the old faulty server is back online, you can start re-integrating it into the infrastructure. Start by enrolling it into the domain:

 [root@ipa01 ~]# ipa-client-install --unattended 
  --principal 'archy' \
  --domain 'archyslife.lan' \
  --server 'ipa03.archyslife.lan' \
  --realm 'ARCHYSLIFE.LAN' \
  --mkhomedir

Once the old node is re-enrolled into the domain, check if you can kinit as the admin user:

 [root@ipa01 ~]# kinit admin  
 Password for admin@ARCHYSLIFE.LAN:

If that worked, you can go ahead and promote the host to a replica:

 [root@ipa01 ~]# ipa hostgroup-add-member ipaservers \
  --hosts 'ipa01.archyslife.lan'  
 [root@ipa01 ~]# ipa-replica-install \
  --setup-ca \
  --setup-dns \
  --setup-kra \
  --forwarder '1.1.1.1' \
  --forwarder '9.9.9.9' \
  --forward-policy 'only'

This process might take a while to complete. After it's done, check if the ipa services are running:

 [root@ipa01 ~]# ipactl status  
 Directory Service: RUNNING  
 krb5kdc Service: RUNNING  
 kadmin Service: RUNNING  
 named Service: RUNNING  
 httpd Service: RUNNING  
 ipa-custodia Service: RUNNING  
 pki-tomcatd Service: RUNNING  
 ipa-otpd Service: RUNNING  
 ipa-dnskeysyncd Service: RUNNING  
 ipa: INFO: The ipactl command was successful

If you want to make sure that this persists a reboot, you can try stopping and starting (not restarting) the services by using the appropriate 'ipactl stop' and 'ipactl start' commands.

The last thing to do is ensure the replication agreements exist. I'll be using the cli:

 [root@ipa01 ~]# ipa topologysegment-add \
  domain 'ipa02.archyslife.lan-to-ipa01.archyslife.lan' \
  --leftnode 'ipa02.archyslife.lan' \
  --rightnode 'ipa01.archyslife.lan'  
 [root@ipa01 ~]# ipa topologysegment-add \
  domain 'ipa03.archyslife.lan-to-ipa01.archyslife.lan' \
  --leftnode 'ipa03.archyslife.lan' \
  --rightnode 'ipa01.archyslife.lan'  
 [root@ipa01 ~]# ipa topologysegment-add \
  ca 'ipa02.archyslife.lan-to-ipa01.archyslife.lan' \
  --leftnode 'ipa02.archyslife.lan' \
  --rightnode 'ipa01.archyslife.lan'  
 [root@ipa01 ~]# ipa topologysegment-add \
  ca 'ipa03.archyslife.lan-to-ipa01.archyslife.lan' \
  --leftnode 'ipa03.archyslife.lan' \
  --rightnode 'ipa01.archyslife.lan'

By now your servers should all be working and replicating with each other according to the replication agreements.

Feel free to comment and / or suggest a topic.

Comments

Dynamic DNS with BIND and ISC-DHCP

I personally prefer to work with hostnames instead of ip-addresses. If you have anything like freeipa or active directory, it will do that for you by registering the client you added to your realm to the managed dns and edit the records dynamically. We can achieve the same goal with just bind and isc-dhcp. I'll use a raspberry pi with raspbian 9 for this setup. So here is a quick tutorial on how to configure the isc-dhcp-server to dynamically update bind. First set a static ip to your server. [archy@ddns ~]$ sudo vim /etc/network/interfaces # interfaces(5) file used by ifup(8) and ifdown(8) # Please note that this file is written to be used with dhcpcd # For static IP, consult /etc/dhcpcd.conf and 'man dhcpcd.conf' # Include files from /etc/network/interfaces.d: source-directory /etc/network/interfaces.d auto eth0 iface eth0 inet static address 172.31.30.5 network 172.31.30.0 broadcast 172.31.30.255 netmask 255.255.255.0 ...

LACP-Teaming on CentOS 7 / RHEL 7

What is teaming? Teaming or LACP (802.3ad) is a technique used to bond together multiple interfaces to achieve higher combined bandwith. NOTE: every clients speed can only be as high as the single link speed of one of the members. That means, if the interfaces I use in the bond have 1 Gigabit, every client will only have a maximum speed of 1 Gigabit. The advantage of teaming is, that it can handle multiple connections with 1 Gigabit. How many connections depends on the amount of your network cards. I'm using 2 network cards for this team on my server. That means I can handle 2 Gigabit connections at full rate on my server provided the rest of the hardware can deliver that speed. There also exists 'Bonding' in the Linux world. They both do the same in theory but for a detailed comparison check out this article about teaming in RHEL7 . To create a teaming-interface, we will first have to remove all the interface configurations we've done on the (soon to be) sla...

Push logs and data into elasticsearch - Part 2 Mikrotik Logs

This is only about the setup of different logging, one being done with Filebeat and the other being done with sending logging to a dedicated port opened in Logstash using the TCP / UDP Inputs. Prerequesites: You'll need a working Elasticsearch Cluster with Logstash and Kibana. Start by getting the Log Data you want to structure parsed correctly. Mikrotik Logs are a bit difficult since they show you Data in the interface which is already enriched with Time / Date. That means a message that the remote logging will send to Logstash will look like this: firewall,info forward: in:lan out:wan, src-mac aa:bb:cc:dd:ee:ff, proto UDP, 172.31.100.154:57061->109.164.113.231:443, len 76 You can check them in the grok debugger and create your own filters and mapping. The following is my example which might not fit your needs. Here are some custom patterns I wrote for my pattern matching: MIKROTIK_DATE \b(?:jan(?:uary)?|feb(?:ruary)?|mar(?:ch)?|apr(?:il)?|may|jun(?:e)?|jul(?...

FreeIPA - Integrating your DHCPD dynamic Updates into IPA

I recently went over my network configuration and noticed that the dhcp-leases were not pushed into the IPA-DNS yet. So I thought, why not do it now. The setup is very similar to setting it up on a single bind instance not managed by IPA (I've already written a guide about this here ). My setup is done with the following hosts: ipa01.archyslife.lan - 172.31.0.1 inf01.archyslife.lan - 172.31.0.5 First of all, create a rndc-key: [archy@ipa01 ~]$ sudo rndc-confgen -a -b 512 This will create the following file '/etc/rndc-key' [archy@ipa01 ~]$ sudo cat /etc/rndc.key key "rndc-key" { algorithm hmac-md5; secret "secret_key_here=="; }; We also need to make named aware of the rndc-key and allow our remote dhcp server to write dns entries: [archy@ipa01 ~]$ sudo vim /etc/named.conf ... include "/etc/rndc-key"; controls { inet 172.31.0.1 port 953 allow { 172.31.0.5; } keys ...

SSSD - Debugging PAM permission denied

Sometimes there's weird errors in IT that occur on random chance. I've had such an encounter with SSSD in combination with IPA(+AD-Trust) recently, where only sometimes, a connection to one of the IPA-Servers would fail with this error: Jul 13 13:36:42 ipa02.archyslife.lan sshd[3478]: pam_sss(sshd:account): Access denied for user runner: 4 (System error) Jul 13 13:36:42 ipa02.archyslife.lan sshd[3478]: fatal: Access denied for user runner by PAM account configuration [preauth] In my case, it was only happening sometimes when running a basic system setup role using ansible on every host in the entire environment. This way, there was no consistent pattern besides being the same host every time if it failed. First up, add the 'debug_level=X' to every section required in the /etc/sssd/sssd.conf where X is a number from 1 to 10 with 10 being the most verbose. Afterward, restart sssd and check the logs for any obvious problems. 1) If you are using local users, check the...

Archy's Blog

Search This Blog