Troubleshooting NetBackup Snapshot Manager for Data Center
Refer to the following troubleshooting scenarios:
Disaster recovery when DR package is lost or passphrase is lost.
This issue may occur if the DR package is lost or the passphrase is lost.
In the case of Catalog backup, two backup packages are created:
DR package which contains all the certs.
Catalog package which contains the database.
The DR package contains the NetBackup UUID cert and Catalog DB also has the UUID. When you perform disaster recovery using the DR package followed by catalog recovery, both the UUID cert and the UUID are restored. This allows NetBackup to communicate with NetBackup Snapshot Manager for Data Center since the UUID is not changed.
However if the DR package is lost or the Passphrase is lost the DR operation cannot be completed. You can only recover the catalog without the DR package after you reinstall NetBackup. In this case, a new UUID is created for NetBackup which is not recognized by NetBackup Snapshot Manager for Data Center. The one-to-one mapping of NetBackup and NetBackup Snapshot Manager for Data Center is lost.
Workaround:
To resolve this issue, you must update the new NetBackup UUID and version number after NetBackup primary is created.
The NetBackup administrator must be logged on to the NetBackup web Management Service to perform this task. Use the following command to log on:
/usr/openv/netbackup/bin/bpnbat -login -loginType WEB
Run the following command on the primary server to get the NetBackup UUID:
/usr/openv/netbackup/bin/admincmd/nbhostmgmt -list -host <primary server host name> | grep "Host ID"
Run the following command to get the version number:
/usr/openv/netbackup/bin/admincmd/bpgetconfig -g <primary Ssrver host name> -L
After you get the NetBackup UUID and Version number, run the following command on the NetBackup Snapshot Manager for Data Center host to update the mapping:
/cloudpoint/scripts/cp_update_nbuuid.sh -i <NBU UUID> -v <Version Number>
When NetBackup Snapshot Manager for Data Center is deployed in private subnet (without internet) NetBackup Snapshot Manager for Data Center function fails
This issue occurs when NetBackup Snapshot Manager for Data Center is deployed in a private network where a firewall is enabled or public IP which is disabled. The customer's information security team would not allow full internet access to the virtual machine's.
Workaround:
Enable the ports from the firewall command line using the following commands:
firewall-cmd --add-port=22/tcp
firewall-cmd --add-port=5671/tcp
firewall-cmd --add-port=443/tcp
After starting the start/stop services, NetBackup Snapshot Manager for Data Center, RabbitMQ and MongoDB containers are still in the starting state
It was observed that flexsnap-mongodb and flexsnap-rabbitmq containers did not go into a healthy state. Following is the state of the flexsnap-mongodb container:
[ec2-user@ip-172-31-23-60 log]$ sudo podman container inspect --format='{{json .Config.Healthcheck}}' flexsnap-mongodb {"Test":["CMD-SHELL","echo 'db.runCommand({ping: 1}).ok' | mongo --ssl --sslCAFile /cloudpoint/keys/cacert.pem --sslPEMKeyFile /cloudpoint/keys/mongodb.pem flexsnap-mongodb:27017/zenbrain --quiet"], "Interval":60,"Timeout":30000000000,"Retries":3} [ec2-user@ip-172-31-23-60 log]$ sudo podman container inspect --format=' {{json .State.Healthcheck}}' flexsnap-mongodb {"Status":"starting","FailingStreak":0,"Log":null} [ec2-user@ip-172-31-23-60 log]$Workaround:
Run the following #podman CLI(s) command:
[ec2-user@ip-172-31-23-60 log]$ sudo podman healthcheck run flexsnap-mongodb [ec2-user@ip-172-31-23-60 log]$ sudo podman ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES fe8cf001032b localhost/veritas/ flexsnap-fluentd:10.0.0.0.9817 2 days ago Up 45 hours ago 0.0.0.0:24224->24224/tcp flexsnap-fluentd 2c00500c1ac6 localhost/veritas/ flexsnap-mongodb:10.0.0.0.9817 2 days ago Up 45 hours ago (healthy) flexsnap-mongodb 7ab3e248024a localhost/veritas/ flexsnap-rabbitmq:10.0.0.0.9817 2 days ago Up 45 hours ago (starting) flexsnap-rabbitmq [ec2-user@ip-172-31-23-60 log]$ sudo podman healthcheck run flexsnap-rabbitmq [ec2-user@ip-172-31-23-60 log]$ sudo podman ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES fe8cf001032b localhost/veritas/ flexsnap-fluentd:10.0.0.0.9817 2 days ago Up 45 hours ago 0.0.0.0:24224->24224/tcp flexsnap-fluentd 2c00500c1ac6 localhost/veritas/ flexsnap-mongodb:10.0.0.0.9817 2 days ago Up 45 hours ago (healthy) flexsnap-mongodb 7ab3e248024a localhost/veritas/ flexsnap-rabbitmq:10.0.0.0.9817 2 days ago Up 45 hours ago (healthy) flexsnap-rabbitmq [ec2-user@ip-172-31-23-60 log]$ sudo podman container inspect --format='{{json .State.Healthcheck}}' flexsnap-mongodb {"Status":"healthy","FailingStreak":0,"Log":[{"Start":"2022-02-14T07:32:13.051150432Z","End":"2022-02-14T07:32:13.444636429Z","ExitCode":0,"Output":""}]} [ec2-user@ip-172-31-23-60 log]$ sudo podman container inspect --format='{{json .State.Healthcheck}}' flexsnap-rabbitmq {"Status":"healthy","FailingStreak":0,"Log":[{"Start":"2022-02-14T07:32:46.537804403Z","End":"2022-02-14T07:32:47.293695744Z","ExitCode":0,"Output":""}]} [ec2-user@ip-172-31-23-60 log]$Certificate generation would fail while registering NetBackup Snapshot Manager for Data Center with NetBackup
Starting NetBackup Snapshot Manager for Data Center release 9.1.2, NetBackup generates the certificates synchronously with registration in the register API of NetBackup Snapshot Manager for Data Center. Hence, any failure in certificate generation causes failure while registering NetBackup Snapshot Manager for Data Center with NetBackup, that is adding or editing the NetBackup Snapshot Manager for Data Center entry from web UI. These certificates are used for datamover which is launched for operations like backup from snapshot, restore from backup, indexing (VxMS based), and so on. Hence, if certificate generation fails, these jobs cannot be performed. Hence NetBackup Snapshot Manager for Data Center on cloud VMs cannot connect to NetBackup on lab VMs, hence the registration will fail, and hence NetBackup Snapshot Manager for Data Center cannot be added to NetBackup.
Workaround:
To add NetBackup Snapshot Manager for Data Center in such a scenario requires to skip certificate generation on NetBackup Snapshot Manager for Data Center by adding the following entry in the
/cloudpoint/flexsnap.conffile:[client_registration] skip_certificate_generation = yes
Plugin information is duplicated, if NetBackup Snapshot Manager for Data Center registration has failed in previous attempts
This occurs only when NetBackup Snapshot Manager for Data Center has been deployed using the MarketPlace Deployment Mechanism. This issue is observed when the plug-in information is added before the registration. This issue creates duplicate plug-in information in the file.
Workaround:
Manually delete the duplicated plug-in information from the file.
For example, consider the following example where the duplicate entry for GCP plug-in config is visible (in bold) in the file:
{ "CPServer1": [ { "Plugin_ID": "test", "Plugin_Type": "aws", "Config_ID": "aws.8dda1bf5-5ead-4d05-912a-71bdc13f55c4", "Plugin_Category": "Cloud", "Disabled": false } ] }, { "CPServer2": [ { "Plugin_ID": "gcp.2080179d-c149-498a-bf1f-4c9d9a76d4dd", "Plugin_Type": "gcp", "Config_ID": "gcp.2080179d-c149-498a-bf1f-4c9d9a76d4dd", "Plugin_Category": "Cloud", "Disabled": false }, ] }Plugin information is duplicated, if cloned NetBackup Snapshot Manager for Data Center is added into NetBackup
This occurs only when cloned NetBackup Snapshot Manager for Data Center is added into NetBackup during migration of NetBackup Snapshot Manager for Data Center to RHEL 8.6 VM. Cloning of NetBackup Snapshot Manager for Data Center uses existing NetBackup Snapshot Manager for Data Center volume to create new NetBackup Snapshot Manager for Data Center. This creates a duplicate entry into the file.
Workaround:
Manually edit and delete the duplicated plug-in information from the file.
For example, consider the following example where the duplicate entry for Azure plug-in config is visible (in bold) in the file:
{ }, { "cpserver101.yogesh.joshi2-dns-zone": [ { "Plugin_ID": "azure.327ec7fc-7a2d-4e94-90a4-02769a2ba521", "Plugin_Type": "azure", "Config_ID": "azure.327ec7fc-7a2d-4e94-90a4-02769a2ba521", "Plugin_Category": "Cloud", "Disabled": false }, { "Plugin_ID": "AZURE_PLUGIN", "Plugin_Type": "azure", "Config_ID": "azure.4400a00a-8d2b-4985-854a-74f48cd4567e", "Plugin_Category": "Cloud", "Disabled": false } ] } ] }