Prevent Xtrabackup Failures for MySQL after Linux OS Patching

Patching the operating system is essential to keeping your server secure and performing optimally. Patching is as easy as running an apt-get update and apt-get upgrade. However, this can cause some unintended consequences. For one, MySQL server will also be upgraded to the latest minor version. This can break processes managed by third party software such as Percona Xtrabackup. Moreover, currently, minor downgrades are not supported.

I re-created the scenario with a VM set up with Ubuntu 20.0.4, MySQL 8.0.28, and Percona Xtrabackup 8.0.12. The screenshot below shows the versions of each.

Prevent Xtrabackup Failures for MySQL after Linux OS Patching apt-get Update and Upgrade

After running the sudo apt-get update and sudo apt-get upgrade commands, MySQL upgraded to 8.0.31 and Percona Xtrabackup did not, as depicted below.

Prevent Xtrabackup Failures for MySQL after Linux OS Patching apt-get Update and Upgrade Did Not Work For Xtrabackup

The next time Xtrabackup tries to perform a backup, you are likely to get an error message like the following one suggesting that Percona Xtrabackup is no longer compatible. The latest error message as followed:

File ./ib_logfile0: 'open' returned OS error 71. Cannot continue operation Cannot continue operation.

Prevent Xtrabackup Failures for MySQL after Linux OS Patching Xtrabackup Error Message

In my experience, there is a high risk of a minor upgrade in MySQL causing the Xtrabackup’s backup process to break. Running the command below will exclude MySQL packages from the update and upgrade process.

$ sudo apt-mark hold mysql-server

As shown by the screenshot below, the hold command prevented the minor upgrade of MySQL.

Prevent Xtrabackup Failures for MySQL after Linux OS Patching Hold Command

Percona will update Xtrabackup as soon as possible. Information on the latest Percona Xtrabackup can be found on their website. Once the latest version is released to address the incompatibilities, you will want to release the hold and install the latest version.

To release the hold, run the following command:

$ sudo apt-mark unhold mysql-server

You can manually upgrade MySQL or use the same sudo apt-get update and sudo apt-get upgrade commands.

If Xtrabackup is not upgraded in the process, run the following commands:

$ wget https://repo.percona.com/apt/percona-release_latest.$(lsb_release -sc)_all.deb
$ sudo dpkg -i percona-release_latest.$(lsb_release -sc)_all.deb

One thing to consider if excluding MySQL from the patching process is the security risk, some serious. Keeping up with the latest patches is always a good idea. Oracle publishes a list of critical patches each quarter on their website, https://www.oracle.com/security-alerts/#CriticalPatchUpdates.

If you have any questions, please don’t hesitate to contact us!