Can't login to box. Error 500!
It can be seen again at night there was an update that did not reach the end or did not fit correctly.
Original question is available on version: ru
Answers:
Good evening.
The problem was not due to an update. The update passed on 10/14/2021 01:19 without any problems.
The server crashed on Oct 14 02:18:01, as you can see from the logs that end at 02:18:01 and only start on Oct 14 09:26:54 when you rebooted the server.
Oct 14 02:17:01 srv65111 systemd: Starting Session 4295 of user root.
Oct 14 02:18:01 srv65111 systemd: Started Session 4296 of user root.
Oct 14 02:18:01 srv65111 systemd: Starting Session 4296 of user root.
Oct 14 09:26:54 srv65111 journal: Runtime journal is using 8.0M (max allowed 189.5M, trying to leave 284.2M free of 1.8G available → current limit 189.5M).
Oct 14 09:26:54 srv65111 kernel: Initializing cgroup subsys cpuset
Oct 14 09:26:54 srv65111 kernel: Initializing cgroup subsys cpu
Oct 14 09:26:54 srv65111 kernel: Initializing cgroup subsys cpuacct
Oct 14 09:26:54 srv65111 kernel: Linux version 3.10.0-862.2.3.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC)) #1 SMP Wed May 9 18:
05:47 UTC 2018
Oct 14 09:26:54 srv65111 kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-862.2.3.el7.x86_64 root=UUID=e00f4793-08de-4f52-b7c4-e73602eb8934 ro crashkernel=auto console=ttyS0, 115200n8
LANG=en_US.UTF-8
Updates are triggered by a watch cron that didn't start after 1:00 am.
The last process on the server started Oct 14 02:18:01 and then only after reboot:
Oct 14 02:16:01 srv65111 CROND[14998]: (root) CMD (/etc/cron.minute.conf.sh > /dev/null 2>&1)
Oct 14 02:17:01 srv65111 CROND[15027]: (root) CMD (/etc/cron.minute.conf.sh > /dev/null 2>&1)
Oct 14 02:18:01 srv65111 CROND[15056]: (root) CMD (/etc/cron.minute.conf.sh > /dev/null 2>&1)
Oct 14 09:27:02 srv65111 crond[498]: (CRON) INFO (RANDOM_DELAY will be scaled with factor 5% if used.)
Oct 14 09:27:02 srv65111 crond[498]: (CRON) INFO (running with inotify support)
Now the system is not working because the file repository and the CRM system engine are damaged.
As far as I understand, you restored the integrity of the database. Because the database tables were also damaged.
The problem was not due to an update. The update passed on 10/14/2021 01:19 without any problems.
The server crashed on Oct 14 02:18:01, as you can see from the logs that end at 02:18:01 and only start on Oct 14 09:26:54 when you rebooted the server.
Oct 14 02:17:01 srv65111 systemd: Starting Session 4295 of user root.
Oct 14 02:18:01 srv65111 systemd: Started Session 4296 of user root.
Oct 14 02:18:01 srv65111 systemd: Starting Session 4296 of user root.
Oct 14 09:26:54 srv65111 journal: Runtime journal is using 8.0M (max allowed 189.5M, trying to leave 284.2M free of 1.8G available → current limit 189.5M).
Oct 14 09:26:54 srv65111 kernel: Initializing cgroup subsys cpuset
Oct 14 09:26:54 srv65111 kernel: Initializing cgroup subsys cpu
Oct 14 09:26:54 srv65111 kernel: Initializing cgroup subsys cpuacct
Oct 14 09:26:54 srv65111 kernel: Linux version 3.10.0-862.2.3.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC)) #1 SMP Wed May 9 18:
05:47 UTC 2018
Oct 14 09:26:54 srv65111 kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-862.2.3.el7.x86_64 root=UUID=e00f4793-08de-4f52-b7c4-e73602eb8934 ro crashkernel=auto console=ttyS0, 115200n8
LANG=en_US.UTF-8
Updates are triggered by a watch cron that didn't start after 1:00 am.
The last process on the server started Oct 14 02:18:01 and then only after reboot:
Oct 14 02:16:01 srv65111 CROND[14998]: (root) CMD (/etc/cron.minute.conf.sh > /dev/null 2>&1)
Oct 14 02:17:01 srv65111 CROND[15027]: (root) CMD (/etc/cron.minute.conf.sh > /dev/null 2>&1)
Oct 14 02:18:01 srv65111 CROND[15056]: (root) CMD (/etc/cron.minute.conf.sh > /dev/null 2>&1)
Oct 14 09:27:02 srv65111 crond[498]: (CRON) INFO (RANDOM_DELAY will be scaled with factor 5% if used.)
Oct 14 09:27:02 srv65111 crond[498]: (CRON) INFO (running with inotify support)
Now the system is not working because the file repository and the CRM system engine are damaged.
As far as I understand, you restored the integrity of the database. Because the database tables were also damaged.
15.10.2021, 00:26
Original comment available on version: ru
Good afternoon
Yes. The database has been restored. (I have backups)
Now it is very necessary to restore the working capacity of the system.
I will be very grateful for your help.
I don't know why, but I started having problems.
On Monday, the system also did not start. The database has been corrupted. We restored it with a backup a day earlier and everything worked.
And now again. But already not only the DB flew off it turns out.
If you have the opportunity to fix the damage. Please do.
Yes. The database has been restored. (I have backups)
Now it is very necessary to restore the working capacity of the system.
I will be very grateful for your help.
I don't know why, but I started having problems.
On Monday, the system also did not start. The database has been corrupted. We restored it with a backup a day earlier and everything worked.
And now again. But already not only the DB flew off it turns out.
If you have the opportunity to fix the damage. Please do.
15.10.2021, 09:01
Original comment available on version: ru
Perhaps it is the SSD on which the system is installed fails?
And the files get corrupted.
And the files get corrupted.
15.10.2021, 09:07
Original comment available on version: ru
Komar Alexander wrote:
Perhaps it is the SSD on which the system is installed fails?
And the files get corrupted.
Good afternoon.
Maybe you are right. We do not monitor client servers, so it is difficult for us to say what is happening on the server and why this is happening.
Komar Alexander wrote:
If you have the opportunity to fix the damage. Please do.
Repair of damaged files - 1 hour.
15.10.2021, 10:52
Original comment available on version: ru
You will be sent an invoice in a letter.
15.10.2021, 11:36
Original comment available on version: ru
Restored from a backup. Many thanks for the help.
15.10.2021, 14:33
Original comment available on version: ru
The cause of the breakdown is not yet clear.
15.10.2021, 14:34
Original comment available on version: ru
Please join the conversation. If you have something to say - please write a comment. You will need a mobile phone and an SMS code for identification to enter.
Log in and comment
Copyright © OneBox 2024
