To resolve backup failure issue in a setup where NetBackup KMS is configured

1. If a NetBackup policy is configured to use tape, AdvanceDisk or cloud storage, check job details. If you see any errors, refer to the NetBackup Status Codes Reference Guide.

   For example in case of tape storage type, you may see the following error in the job details tab:

   Mar 27, 2020 5:20:40 PM - Error bptm (pid=11143) KMS failed with error status: Error details : 
   Error Code : 1298, Error Message : Cannot communicate with one or more key management servers., 
   Server - example.master.com:0, Error code - 25, .    
   Mar 27, 2020 5:20:40 PM - Info bptm (pid=11143) EXITING with status 83 <----------    
   Mar 27, 2020 5:20:43 PM - Info bpbkar (pid=11132) done. status: 83: media open error

2. Run the following command on the master server to verify whether NetBackup KMS is configured or not:

   Install_Path/bin/nbkmscmd -listKMSConfig -name nbkms

   If NetBackup KMS configuration is not listed, check if the nbkms service is running or not.

   • If the nbkms service is running, run the following command to add the nbkms service configuration:

     Install_Path/bin/nbkmscmd -discoverNBkms

   • If nbkms service is not running check nbkms logs at the following location:

     On UNIX - /usr/openv/logs/nbkms

     On Windows - Install_Path\NetBackup\logs\nbkms

     Check if a key is created on the KMS server with the required key group.

3. Validate the NetBackup KMS configuration using the following command:

   Install_Path/bin/nbkmscmd -validateKMSConfig -name KMS_configuration_name

4. Check if at least one active key is listed using the following command:

   Install_Path/bin/nbkmscmd -listKeys -name KMS_configuration_name -keyGroupName key_group_name

5. If key is not listed, create a key with the required key group and clear the cache on the media server. Run the following command:

   Install_Path/bin/bpclntcmd -clear_host_cache

6. Check the following logs for further details:

   In case of tape, AdvanceDisk, and cloud storage: Install_Path/netbackup/logs/bptm

   In case of MSDP storage: MSDP_config_path/log/spoold/spoold.log

   For web service logs on the master server: Install_path/logs/nbwebservice/<51216-495-***-***-***.log>

   For nbkmiputil logs for NetBackup KMS: Install_Path/logs/nbkms

To resolve backup failure issue in a setup where external KMS is configured

1. If a NetBackup policy is configured to use tape, AdvanceDisk or cloud storage, check job details. If you see any errors, refer to the NetBackup Status Codes Reference Guide.
2. Run the following command on the master server to verify whether external KMS is configured or not:

   Install_Path/bin/nbkmscmd -listKMSConfig -name KMS_configuration_name

   If configuration is not listed, configure external KMS server.

3. Validate the external KMS configuration using the following command:

   Install_Path/bin/nbkmscmd -validateKMSConfig -name KMS_configuration_name

4. Run the following command if certificate files exist on the master server.

   Install_Path/netbackup/bin/goodies/nbkmiputil -validate -kmsServer kms_server_name -port 5696 -certPath certificate_file_path -privateKeyPath private_key__file_path -trustStorePath ca_file_path

   The output is in a JSON format.

5. Check if key is created on external KMS server with the required key group.
6. Check if at least one active key is listed using the following command:

   Install_Path/bin/nbkmscmd -listKeys -name KMS_configuration_name -keyGroupName key_group_name

   If key is not listed, create a key with the required key group and clear the cache on the media server. Run the following command:

   Install_Path/bin/bpclntcmd -clear_host_cache

7. Check the following logs for further details:

   In case of tape, AdvanceDisk, and cloud storage: Install_Path/netbackup/logs/bptm

   In case of MSDP storage: PDDE_Install_Path/log/spoold/spoold.log

   For web service logs on the master server: Install_Path/logs/nbwebservice/<51216-495-***-***-***.log>

   For nbkmiputil logs for external KMS:Install_Path/netbackup/logs/nbkmiputil

To resolve restore failure issue

1. In case of tape, AdvanceDisk, and cloud storage, check job details.

2. Validate the KMS configuration using the following commands:

   Install_Path/bin/nbkmscmd -validateKMSConfig -name KMS_configuration_name

3. Run the following command if certificate files exist on master server, Install_Path/netbackup/bin/goodies/nbkmiputil -validate -kmsServer KMS_server_name -port 5696 -certPath certificate_file_path -privateKeyPath private_key__file_path -trustStorePath ca_file_path

   The output is displayed in the JSON format.

4. Ensure that the key with which backup is encrypted is still active on the KMS server.

   See the following error in nbwebservice logs to get the key tag that is required for restore.

   See the following log statements in the web service logs on the master server: Install_path/logs/nbwebservice/<51216-495-***-***-***.log>

   Here are the log snippets:

   [Debug] NB 51216 nbwebapi 495 PID:10984 TID:149 File ID:495 [No context] 5 
   	[com.netbackup.config.PeerInfoPopulatorFilter] 
   Request URL : https://<Master-Server>:1556/netbackup/security/key-management-services/keys 
   Connection Info :ConnectionInfo

   [Debug] NB 51216 nbwebapi 495 PID:10984 TID:149 File ID:495 [No context] 5 
   [com.netbackup.security.kms.resource.KMSConfigResource] 
   HTTP GET filter query string is : KeyId eq 'bdc3492b015d4a9ab25426465b12adac6a834dfc6b4449c490922d6155719958' 
    and kadlen eq 32

   [Debug] NB 51216 nbwebapi 495 PID:10984 TID:149 File ID:495 [No context] 5 
   [com.netbackup.security.kms.resource.KMSConfigResource] 
   com.netbackup.security.kms.resource.KMSConfigResource getKeys() - NBKMSRecordNotFoundException 
   occured due to missing KMS record.com.netbackup.nbkms.exception.NBKMSRecordNotFoundException: 
   security.error.kms.KeyRecordNotFound

5. Check the following logs for further details:

   For tape, AdvanceDisk, and cloud storage: Install_Path/netbackup/logs/bptm

   For MSDP storage: PDDE_Install_Path/log/spoold/spoold.log

   For web service logs on master server: Install_Path/logs/nbwebservice/<51216-495-***-***-***.log>

   For nbkmiputil logs:

   • For NetBackup KMS,Install_Path/logs/nbkms

   • For external KMS,Install_Path/netbackup/logs/nbkmiputil