Skip to main content

Cron jobs have stopped working due to PAM [Resolved]

My cron jobs have stopped working on my CentOS 7 server. The server is running WHM/cPanel.

It seems like it is an issue with PAM service because in /var/log/secure I can see the following errors when the cron jobs try to run:

Jun 24 10:45:01 server1 crond[22400]: pam_access(crond:account): auth could not identify password for [root]
Jun 24 10:45:01 server1 crond[22404]: pam_access(crond:account): auth could not identify password for [admin]
Jun 24 10:45:01 server1 crond[22400]: pam_localuser(crond:account): auth could not identify password for [root]
Jun 24 10:45:01 server1 crond[22402]: pam_access(crond:account): auth could not identify password for [root]
Jun 24 10:45:01 server1 crond[22405]: pam_access(crond:account): auth could not identify password for [admin]
Jun 24 10:45:01 server1 crond[22400]: pam_localuser(crond:account): auth could not identify password for [root]

Similarly /var/log/cron.log is showing that PAM is failing:

Jun 24 12:40:01 server1 crond[26129]: (admin) PAM ERROR (Authentication information cannot be recovered)
Jun 24 12:40:01 server1 crond[26129]: (admin) FAILED to authorize user with PAM (Authentication information cannot be recovered)
Jun 24 12:40:01 server1 crond[26130]: (admin) PAM ERROR (Authentication information cannot be recovered)
Jun 24 12:40:01 server1 crond[26130]: (admin) FAILED to authorize user with PAM (Authentication information cannot be recovered)
Jun 24 12:40:01 server1 crond[26125]: (root) PAM ERROR (Authentication information cannot be recovered)
Jun 24 12:40:01 server1 crond[26125]: (root) FAILED to authorize user with PAM (Authentication information cannot be recovered)
Jun 24 12:40:01 server1 crond[26127]: (root) PAM ERROR (Authentication information cannot be recovered)
Jun 24 12:40:01 server1 crond[26127]: (root) FAILED to authorize user with PAM (Authentication information cannot be recovered)
Jun 24 12:40:01 server1 crond[26131]: (admin) PAM ERROR (Authentication information cannot be recovered)
Jun 24 12:40:01 server1 crond[26131]: (admin) FAILED to authorize user with PAM (Authentication information cannot be recovered)

I've tried the following with no success:

  • Rebooting the server
  • Restarting the cron service
  • Editing /etc/security/access.conf to ensure that root is allowed access to the cron
  • cron.allow is non-existent and cron.deny is empty so that shouldn't be the problem
  • Disabling SELinux and rebooting
  • Changing root password to ensure it's not an expiry issue
  • Checked /etc/passwd and /etc/shadow for passwords of both root and admin user
  • Removed all cron jobs except a simple one to write to a text file every minute. This cron job also did not work so it's not related to the jobs in the cron.

Please help as I'm not sure what else to do to fix this problem. It's worth noting that this issue started on June 21 and no changes were made to the server when it began occurring.


Question Credit: pauloz1890
Question Reference
Asked July 20, 2019
Tags: centos, cron
Posted Under: Unix Linux
11 views
2 Answers

I was able to solve the issue. It is related to a file called /lib/libgrubd.so.

If you're experiencing this issue then check /etc/ld.so.preload. If this file contains /lib/libgrubd.so (it may be the only line in that file) then remove that line and PAM should start working again. I also removed the /lib/libgrubd.so file from the system as it may be associated with a virus as shown here.

Still not entirely sure what caused the issue but this was the reason causing PAM to functioning incorrectly. See more info here.


credit: pauloz1890
Answered July 20, 2019

I just faced the same issue. What they tell me is this:

I regret to inform you that this server is compromised at the root level by malware known as ShellBot. This malware is known to cause errors when running the "crontab" command but can potentially cause many other problems including not being able to start certain services.

The presence of /lib/libgrubd.so (which is not normally found on clean systems) is an indication of this malware. This malware is installed in a way that causes it to execute every time a program is run on the server, and can potentially cause aberrant behavior of any process.


credit: Jeff Schaller
Answered July 20, 2019
Your Answer
D:\Adnan\Candoerz\CandoProject\vQA