Primary server corrupted
When catalog backup is taken on external media server
When catalog backup is taken on MSDP-X
- Copy DRPackages files (packages) located at
/mnt/nblogs/DRPackages/from the pod to the host machine from where Kubernetes Service cluster is accessed.Run the kubectl cp <primary-pod-namespace>/<primary-pod-name>:/mnt/nblogs/DRPackages <Path_where_to_copy_on_host_machine> command.
- Preserve the data of
/mnt/nbdataand/mnt/nblogson host machine by creating tar and copying it using the kubectl cp <primary-pod-namespace>/<primary-pod-name>:<tar_file_name> <path_on_host_machine_where_to_preserve_the_data> command. - Change CR spec from paused: false to paused: true in primary, mediaServers, and msdpScaleouts sections in environment object using the following command:
kubectl edit <environment_CR_name> -n <namespace>
- Change replica count to 0 in primary server's statefulset using the kubectl edit statefulset <primary-server-statefulset-name> -n <namespace> command.
- Clean the PV and PVCs of primary server as follows:
Get names of PV attached to primary server PVC (catalog, log and data) using the kubectl get pvc -n <namespace> -o wide command.
Delete primary server PVC (catalog, log and data) using the kubectl delete pvc <pvc-name> -n <namespace> command.
Delete the PV linked to primary server PVC using the kubectl delete pv <pv-name> command.
- (EKS-specific) Navigate to mounted EFS directory and delete the content from primary_catalog folder by running the rm -rf /efs/* command.
- Change CR spec paused: true to paused: false in primary server section in and reapply yaml with the kubectl apply -f environment.yaml -n <namespace> command.
- Once the primary pod is in ready state, execute the following command in the primary server pod:
kubectl exec -it -n <namespace> <primary-pod-name> -- /bin/bash
Increase the debug logs level on primary server.
Create a directory
DRPackagesat persisted location usingmkdir /mnt/nblogs/DRPackages.Change ownership of the
DRPackagesfolder to service user using the chown nbsvcusr:nbsvcusr /mnt/nblogs/DRPackages command.
- Copy earlier copied DR files to primary pod at
/mnt/nblogs/DRPackagesusing the kubectl cp <Path_of_DRPackages_on_host_machine> <primary-pod-namespace>/<primary-pod-name>:/mnt/nblogs/DRPackages command. - (Applicable for catalog backup taken on external media server)
Execute the following steps in the primary server pod:
Change ownership of files in
/mnt/nblogs/DRPackagesusing the chown nbsvcusr:nbsvcusr <file-name> command.Deactivate NetBackup health probes using the /opt/veritas/vxapp-manage/nb-health deactivate command.
Stop the NetBackup services using
/usr/openv/netbackup/bin/bp.kill_all.Execute the nbhostidentity -import -infile /mnt/nblogs/DRPackages/<filename>.drpkg command.
Restart all the NetBackup services using
/usr/openv/netbackup/bin/bp.start_all.
Verify security settings are back.
Add respective media server entry in host properties using NetBackup Administration Console as follows:
Navigate to NetBackup Management > Host properties > Master Server > Add Additional server and add media server.
Restart the NetBackup services in primary server pod and external media server
Execute the kubectl exec -it -n <namespace> <primary-pod-name> -- /bin/bash command in the primary server pod.
Run the /usr/openv/netbackup/bin/bp.kill_all command. After stopping all services restart the same using the /usr/openv/netbackup/bin/bp.start_all command.
Run the /usr/openv/netbackup/bin/bp.kill_all command. After stopping all services restart the services using the /usr/openv/netbackup/bin/bp.start_all command on the external media server.
Perform catalog recovery from NetBackup Administration Console.
For more information, refer to the NetBackup Troubleshooting Guide.
Execute the kubectl exec -it -n <namespace> <primary-pod-name> -- /bin/bash command in the primary server pod.
Stop the NetBackup services using the /usr/openv/netbackup/bin/bp.kill_all command.
Start NetBackup services using the /usr/openv/netbackup/bin/bp.start_all command.
Activate NetBackup health probes using the /opt/veritas/vxapp-manage/nb-health activate command.
Delete the currently running request router pod using the following command:
kubectl delete pod <request-router-pod-name> -n <PrimaryServer-namespace>
Change CR spec from paused: true to paused: false in primary, mediaServers, and msdpScaleouts sections in environment object using the following command:
kubectl edit <environment_CR_name> -n <namespace>
To configure NetBackup IT Analytics refer to the following topic:
See Configuring NetBackup IT Analytics for NetBackup deployment.
- (Applicable for catalog backup taken on MSDP-X)
Execute the following steps (after exec) into the primary server pod:
Change ownership of files in
/mnt/nblogs/DRPackagesusing the chown nbsvcusr:nbsvcusr <file-name> command.Deactivate NetBackup health probes using the /opt/veritas/vxapp-manage/nb-health deactivate command.
Stop the NetBackup services using the /usr/openv/netbackup/bin/bp.kill_all command.
Execute the /usr/openv/netbackup/bin/admincmd/nbhostidentity -import -infile /mnt/ndbdb/usr/openv/drpackage/<filename>.drpkg command.
Clear bpclntcmd -clear_host_cacheNetBackup host cache by running the command.
Start NetBackup services using the /usr/openv/netbackup/bin/bp.start_all command.
Refresh the certificate revocation list using the /usr/openv/netbackup/bin/nbcertcmd -getcrl command.
From Web UI, allow reissue of token from primary server for MSDP only as follows:
Navigate to Security > Host Mappings for the MSDP storage server and select Allow Auto reissue Certificate.
Run the primary server reconciler as follows:
Edit the environment (using kubectl edit environment -n <namespace> command) and change primary spec's for paused field to true and save it.
To enable the reconciler to run, the environment must be edited again and the primary's paused field must be set to false.
The SHA fingerprint is updated in the primary CR's status.
Edit the environment using kubectl edit environment -n <namespace> command and change paused field to false for MSDP.
Verify if MSDP installation is successful and default MSDP storage server, STU and disk pool is created with old names. This takes some time. Hence, wait before the STU and disk pool display on the Web UI before proceeding to the next step.
Perform from step 2 in the following section:
Edit environment CR and change
paused: falsefor media server.Perform full catalog recovery using one of the following options:
Trigger a catalog recovery from the Web UI.
Or
Exec into primary pod and run bprecover -wizard command.
Once recovery is completed, restart the NetBackup services:
Stop NetBackup services using the /usr/openv/netbackup/bin/bp.kill_all command.
Start NetBackup services using the /usr/openv/netbackup/bin/bp.start_all command.
Activate NetBackup health probes using the /opt/veritas/vxapp-manage/nb-health activate command.
Verify/Backup/Restore the backup images in NetBackup server to check if the MSDP-X cluster has recovered or not.
Verify that the Primary, Media, MSDP and Snapshot Manager server are up and running.