SSH - Debugging Authentication Failures

Following a server reboot, I encountered an issue where the rpcbind.service failed to start. Manually restarting the service resulted in subsequent login failures, as indicated by the following error:

 System is booting up. Unprivileged users are not permitted to log in yet. Please come back later. For technical details, see pam_nologin(8).   

Here's a record of how I approached debugging this issue. I'm sharing it in case my experience can help others learn and troubleshoot similar problems.

The server is joined to FreeIPA and HBAC rules permit access, as confirmed by the following 'ipa hbactest' output:

 [root@admin03 ~]# ipa hbactest --user archy --host admin03.archyslife.lan --service sshd | grep -viIE 'not'  
 --------------------  
 Access granted: True  
 --------------------  
  Matched rules: allow-admin-users-admin-hosts  

Reviewing the /var/log/secure log showed a PAM account permission issue that was blocking user logins:

 Feb 17 18:23:26 admin03 sshd[14485]: User archy authorized keys /dev/null is not a regular file  
 Feb 17 18:23:26 admin03 sshd[14485]: Accepted key ED25519 SHA256:${SSHKEY_FINGERPRINT} found at /usr/bin/sss_ssh_authorizedkeys:1  
 Feb 17 18:23:26 admin03 sshd[14485]: Failed publickey for archy from ${CLIENT_IP} port 36202 ssh2: ED25519 SHA256:${SSHKEY_FINGERPRINT}  
 Feb 17 18:23:26 admin03 sshd[14485]: fatal: Access denied for user archy by PAM account configuration [preauth]  

The logs suggested a problem with the PAM configuration for SSH. I then reviewed the contents of /etc/pam.d/sshd:

 [root@admin03 ~]# cat /etc/pam.d/sshd  
 #%PAM-1.0  
 auth    substack   password-auth  
 auth    include   postlogin  
 account  required   pam_sepermit.so  
 account  required   pam_nologin.so  
 account  include   password-auth  
 password  include   password-auth  
 # pam_selinux.so close should be the first session rule  
 session  required   pam_selinux.so close  
 session  required   pam_loginuid.so  
 # pam_selinux.so open should only be followed by sessions to be executed in the user context  
 session  required   pam_selinux.so open env_params  
 session  required   pam_namespace.so  
 session  optional   pam_keyinit.so force revoke  
 #session  optional   pam_motd.so  
 session  include   password-auth  
 session  include   postlogin  

The /etc/pam.d/sshd configuration included pam_nologin.so, prompting me to examine the module's documentation. The man page provided this information:

 OPTIONS  
     file=/path/nologin  
       Use this file instead the default /var/run/nologin or /etc/nologin.  

Let's check if that file exists - it could be the root cause of the login failures:

 [root@admin03 ~]# if [ -f /run/nologin ]; then echo -e 'file exists'; fi  
 file exists  

The file is present. I will now remove it and then retest the login process:

 [root@admin03 ~]# rm -f /run/nologin  

The login works It's also worth checking the state of these systemd units:

 [root@admin03 ~]# systemctl status systemd-tmpfiles-setup.service  
 [root@admin03 ~]# systemctl status systemd-user-sessions.service  

Make sure, they're enabled and started properly since these two services are involved in making sure the '/run/nologin' file is removed after starting up.

Feel free to comment and / or suggest a topic.