Troubleshooting issues with KMS configuration
NetBackup supports NetBackup Key Management Service (NetBackup KMS) and external key management service (external KMS).
This section provides procedures to resolve the backup failure issue in the following scenarios:
When NetBackup KMS is configured
When external KMS is configured
See the NetBackup Security and Encryption Guide for more information about KMS configurations.
To resolve backup failure issue in a setup where NetBackup KMS is configured
- If a NetBackup policy is configured to use tape, AdvanceDisk or cloud storage, check job details. If you see any errors, refer to the NetBackup Status Codes Reference Guide.
For example in case of tape storage type, you may see the following error in the job details tab:
Mar 27, 2020 5:20:40 PM - Error bptm (pid=11143) KMS failed with error status: Error details : Error Code : 1298, Error Message : Cannot communicate with one or more key management servers., Server - example.primary.com:0, Error code - 25, . Mar 27, 2020 5:20:40 PM - Info bptm (pid=11143) EXITING with status 83 <---------- Mar 27, 2020 5:20:43 PM - Info bpbkar (pid=11132) done. status: 83: media open error
- Run the following command on the primary server to verify whether NetBackup KMS is configured or not:
Install_Path/bin/nbkmscmd -listKMSConfig -name nbkms
If NetBackup KMS configuration is not listed, check if the nbkms service is running or not.
If the nbkms service is running, run the following command to add the nbkms service configuration:
Install_Path/bin/nbkmscmd -discoverNBkms
If nbkms service is not running check nbkms logs at the following location:
On UNIX - /usr/openv/logs/nbkms
On Windows - Install_Path\NetBackup\logs\nbkms
Check if a key is created on the KMS server with the required key group.
- Validate the NetBackup KMS configuration using the following command:
Install_Path/bin/nbkmscmd -validateKMSConfig -name KMS_configuration_name
- Check if at least one active key is listed using the following command:
Install_Path/bin/nbkmscmd -listKeys -name KMS_configuration_name -keyGroupName key_group_name
- If key is not listed, create a key with the required key group and clear the cache on the media server. Run the following command:
Install_Path/bin/bpclntcmd -clear_host_cache
- Check the following logs for further details:
In case of tape, AdvanceDisk, and cloud storage: Install_Path/netbackup/logs/bptm
In case of MSDP storage: MSDP_config_path/log/spoold/spoold.log
For web service logs on the primary server: Install_path/logs/nbwebservice/<51216-495-***-***-***.log>
For nbkmiputil logs for NetBackup KMS: Install_Path/logs/nbkms
To resolve backup failure issue in a setup where external KMS is configured
- If a NetBackup policy is configured to use tape, AdvanceDisk or cloud storage, check job details. If you see any errors, refer to the NetBackup Status Codes Reference Guide.
- Run the following command on the primary server to verify whether external KMS is configured or not:
Install_Path/bin/nbkmscmd -listKMSConfig -name KMS_configuration_name
If configuration is not listed, configure external KMS server.
- Validate the external KMS configuration using the following command:
Install_Path/bin/nbkmscmd -validateKMSConfig -name KMS_configuration_name
- Run the following command if certificate files exist on the primary server.
Install_Path/netbackup/bin/goodies/nbkmiputil -validate -kmsServer kms_server_name -port 5696 -certPath certificate_file_path -privateKeyPath private_key__file_path -trustStorePath ca_file_path
The output is in a JSON format.
- Check if key is created on external KMS server with the required key group.
- Check if at least one active key is listed using the following command:
Install_Path/bin/nbkmscmd -listKeys -name KMS_configuration_name -keyGroupName key_group_name
If key is not listed, create a key with the required key group and clear the cache on the media server. Run the following command:
Install_Path/bin/bpclntcmd -clear_host_cache
- Check the following logs for further details:
In case of tape, AdvanceDisk, and cloud storage: Install_Path/netbackup/logs/bptm
In case of MSDP storage: PDDE_Install_Path/log/spoold/spoold.log
For web service logs on the primary server: Install_Path/logs/nbwebservice/<51216-495-***-***-***.log>
For nbkmiputil logs for external KMS:Install_Path/netbackup/logs/nbkmiputil
Use the following procedure to resolve the restore failure issue in case of a storage that is KMS enabled:
To resolve restore failure issue
- In case of tape, AdvanceDisk, and cloud storage, check job details.
- Validate the KMS configuration using the following commands:
Install_Path/bin/nbkmscmd -validateKMSConfig -name KMS_configuration_name
- Run the following command if certificate files exist on primary server, Install_Path/netbackup/bin/goodies/nbkmiputil -validate -kmsServer KMS_server_name -port 5696 -certPath certificate_file_path -privateKeyPath private_key__file_path -trustStorePath ca_file_path
The output is displayed in the JSON format.
- Ensure that the key with which backup is encrypted is still active on the KMS server.
See the following error in nbwebservice logs to get the key tag that is required for restore.
See the following log statements in the web service logs on the primary server: Install_path/logs/nbwebservice/<51216-495-***-***-***.log>
Here are the log snippets:
[Debug] NB 51216 nbwebapi 495 PID:10984 TID:149 File ID:495 [No context] 5 [com.netbackup.config.PeerInfoPopulatorFilter] Request URL : https://<Primary-Server>:1556/netbackup/security/key-management-services/keys Connection Info :ConnectionInfo
[Debug] NB 51216 nbwebapi 495 PID:10984 TID:149 File ID:495 [No context] 5 [com.netbackup.security.kms.resource.KMSConfigResource] HTTP GET filter query string is : KeyId eq 'bdc3492b015d4a9ab25426465b12adac6a834dfc6b4449c490922d6155719958' and kadlen eq 32
[Debug] NB 51216 nbwebapi 495 PID:10984 TID:149 File ID:495 [No context] 5 [com.netbackup.security.kms.resource.KMSConfigResource] com.netbackup.security.kms.resource.KMSConfigResource getKeys() - NBKMSRecordNotFoundException occured due to missing KMS record.com.netbackup.nbkms.exception.NBKMSRecordNotFoundException: security.error.kms.KeyRecordNotFound
- Check the following logs for further details:
For tape, AdvanceDisk, and cloud storage: Install_Path/netbackup/logs/bptm
For MSDP storage: PDDE_Install_Path/log/spoold/spoold.log
For web service logs on primary server: Install_Path/logs/nbwebservice/<51216-495-***-***-***.log>
For nbkmiputil logs:
For NetBackup KMS,Install_Path/logs/nbkms
For external KMS,Install_Path/netbackup/logs/nbkmiputil