Linux OOM Killer Terminated mysqld and How to Recover

by | Jul 8, 2025 | Blog, MySQL

One morning, I was alerted that a crash had been detected by our monitoring scripts. Upon reviewing the error log, a crash was noted.

Error log:
250608 02:27:07 mysqld_safe Number of processes running now: 0
250608 02:27:07 mysqld_safe mysqld restarted
. . . . . . . . . . . . . . . . . . . . . . .
InnoDB: Starting crash recovery.

If you’re running MySQL and encounter a restart followed by crash recovery messages, such as the one below, it’s essential to investigate quickly and carefully.

The error log tells us a few things:

  • mysqld_safe noticed that mysqld was no longer running (i.e., it crashed or was terminated).
  • It immediately restarted the server.
  • Since MySQL did not shut down properly, it now starts crash recovery to replay unflushed transactions from the redo logs.

To better identify the root cause, I checked the system logs:

sudo dmesg | grep -i mysql
sudo journalctl -xe | grep mysql

or /var/log/messages, /var/log/syslog, etc.

The output from dmesg was informative:

[ 4626]     0  4626    27084      167   5       0             0 mysqld_safe
[ 5105]   496  9105  5692133  4327323   0       0             0 mysqld
Out of memory: Kill process 5105 (mysqld) score 530 or sacrifice child
Killed process 9105, UID 496, (mysqld) total-vm:22768532kB, anon-rss:17305284kB, file-rss:4008kB

The output told me the following:

  • The Linux OOM killer terminated the mysqld process (PID 5105) because the system ran out of memory.
  • mysqld was using ~17 GB of anonymous resident memory (anon-rss) and had a very high OOM score (530).
  • UID 496 corresponds to the MySQL user, as expected.

To summarize, MySQL crashed because it was consuming too much memory, and the Linux kernel forcibly killed it to protect the system. That’s why mysqld_safe later restarted it, and InnoDB went into crash recovery mode.

  1. Reduce memory usage in MySQL. Evaluate the buffer settings for  innodb_buffer_pool_size and innodb_log_buffer_size. Good tools to use for analysis are mysqltuner.pl or pt-mysql-summary.
  2. Add or reserve system memory. Set up a swap space if one does not already exist. If possible, increase system RAM or limit other processes that compete for memory.
  3. Prevent future OOM kills. Consider protecting MySQL from OOM.

Keep your database top-notch with XTIVIA’s expert MySQL DBA support and managed services.

For more information, please contact us.