Terminating Db2 Backups to a Vendor Media Server

Sometimes, while running a database backup or restore, you may encounter a situation where you need to terminate the backup or restore operation. If the backup is writing to disk, this is quite easy: You can generally type Ctrl-C, and the Db2 Command Line Processor will stop the backup.

However, if your backup (or restore) is going to a vendor media server, such as NetBackup or TSM, typing Ctrl-C will not work. It does not work because the Db2 media controller EDU is executing code from a 3rd party library; to protect the Db2 engine from bugs in this 3rd party code, the code is executed in a separate process from the Db2 engine. These processes are called Fenced Vendor Processes, and you will see separate db2vend processes running on the server while your backups are active. For a number of years, there was no mechanism for the Db2 agent executing the backup to send send a signal to the 3rd party code running inside of the fenced vendor process and terminate the active backup.

In the past, I often had to call the team responsible for the media server and request that they perform the equivalent of a FORCE APPLICATION on the media server’s console in order to terminate the Db2 backup.

db2pd to the Rescue

In Db2 9.5, IBM added the -fvp option to db2pd. This switch returns information about the Fenced Vendor Processes, and IBM later added a term option to -fvp that sends a signal to the vendor library, causing it to terminate the backup or restore.

Unfortunately, the documentation for how to do this is not entirely clear. Running db2pd -help provides the following description for -fvp:

-fvp { | LAM1 | LAM2 | LAM3} [term] [file=filename]
   Fenced Vendor Process Info

The LAM1, LAM2 and LAM3 options are used for log archiving. For backups, we need to use the agent EDU. The Knowledge Center provides the following information:

Unfortunately, the Knowledge Center does not describe _how_ to find the agent eduid for the backup. Another section of the db2pd documentation suggests using db2pd -edus, but provides no information on how to determine which specific EDU is the agent EDU for the backup. The page also suggests looking in the db2diag.log, but provides no information about what entry you should look for.

I have not found a way to determine which is the correct EDU from the output of db2pd -edus, but I have determined how to find the agent EDU from the db2diag.log. Using the following db2diag command, you should get an entry similar to what is shown below:

$ db2diag -gi function:=sqludPrintStartingMsg

2019-11-22-13.45.49.308875-480 E19986879E559 LEVEL: Info
PID : 30465 TID : 140071323821824 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : SAMPLE
APPHDL : 0-149 APPID: *LOCAL.db2inst1.191122214543
AUTHID : DB2INST1 HOSTNAME: db2server
EDUID : 299 EDUNAME: db2agent (SAMPLE) 0
FUNCTION: DB2 UDB, database utilities, sqludPrintStartingMsg, probe:1464
DATA #1 : Starting a full database restore.
Agent EDU ID: 27

In this case, we are interested in the Agent EDU ID 27.

To get information about the fenced vendor processes, we can run:

$ db2pd -db sample -fvp 27

List of fenced vendor processes

PID Name
===========================================================
31232 db2vend (db2med - 297 (SAMPLE))
30475 db2vend (PD Vendor Process - 1)

-------------------------------------------------------------------------
Fenced Vendor Process State Information:
-------------------------------------------------------------------------

Restore:
-------------------------------------------------------------------------

Media Controller(s):
-------------------------------------------------------------------------
EDU ID: 297
mediaSession: 1
Vendor EDU is available and running.
startTime: 20191122134226
function: sqluvget

And finally, to terminate the backup, you can simply add the `term` option to the previous command:

$ db2pd -db sample -fvp 27 term

List of fenced vendor processes

PID Name
===========================================================
31232 db2vend (db2med - 297 (SAMPLE))
30475 db2vend (PD Vendor Process - 1)

-------------------------------------------------------------------------
Fenced Vendor Process State Information:
-------------------------------------------------------------------------

Restore:
-------------------------------------------------------------------------

Media Controller(s):
-------------------------------------------------------------------------
EDU ID: 297
mediaSession: 1
Vendor EDU is available and running.
startTime: 20191122134229
function: sqluvget
This fenced vendor process has been sent a signal to terminate.

It may take a few minutes for the backup or restore operation to terminate, but eventually, the backup or restore operation will return an error:

$ db2 "restore database sample load /usr/openv/netbackup/bin/nbdb2.so64 \
taken at 20191122094641"
SQL2079N An error was reported by the shared library
"/usr/openv/netbackup/bin/nbdb2.so64". Return code: "30".

SQL2079N Return code 30 indicates that “A severe error was experienced inside the vendor product”, but of course, this is expected because of the way that the backup or restore was stopped.