Troubleshooting issues for kubectl plugin
If the primary key is manually deleted by editing the secrets/sp-keys in the environment, the key is not automatically recreated by the system. This results in missing or invalid service principal keys, which can affect environment operations.
Workaround:
Perform the following steps to manually recreate the deleted key:
Pause the environment reconciler using helm command to set the following parameter:
paused: true
Delete the existing secret by running the following command:
- kubectl delete secrets/sp-keys -n netbackup
Delete all the API keys using the following API's call with a valid JWT token:
Trigger GET netbackup/security/service-principal-configs API.
Find all the service principal configurations where and note the value of for all these service principal configurations.
Trigger DELETE security/service-principal-configs/{id} API for each captured above.
Restart the operator pod by deleting the operator pod:
- kubectl delete pod <operator-pod-name> -n netbackup-operator-system
Un-pause the environment reconciler using helm command to set the following parameter:
paused: false
During the Cloud Scale installation, if the process breaks or stops midway, the deployment does not complete successfully and must be resumed from the point of failure.
Workaround:
To resume the installation process:
Run the following command:
kubectl-cloudscale install
When prompted to resume the installation, type to continue from where it stopped.
The plugin will automatically read the previously saved inputs and resume the installation from the point of interruption.
If a user enters an incorrect Cloud Scale folder path or provides any wrong input during the upgrade process, there is no option to correct it. Re-triggering the plugin simply skips to the next question without allowing the user to re-enter inputs.
Workaround:
To reset and re-enter all upgrade inputs:
Delete the following file:
rm /home/<user-name>/kubectl-plugin/upgrade.csv
Re-run the upgrade command:
kubectl-cloudscale upgrade
The plugin will prompt for all required user inputs again.
Following is an example of the logs:
*******************************Checking for cert-manager******************************** {Component: Installation of helm chart, ComponentName: jetstack/cert-manager} INFO: 2025/09/26 12:38:32 logger.go:132: Checking if Cert Manager is installed or not {Component: Get pod status, ComponentName: Dependency of Cloudscale component} INFO: 2025/09/26 12:38:32 logger.go:132: Waiting 10 seconds before retrying... {Component: Get pod status, ComponentName: Dependency of Cloudscale component} INFO: 2025/09/26 12:38:50 logger.go:132: Cloudscale Upgrade Before you proceed with upgrade, please ensure the following prerequisites are in place: 1. Infrastructure readiness: - The Kubernetes cluster is up and running - Cloudscale environment is up and running 2. Required container images: - All Cloudscale-related images of the version you would like to upgrade to are pushed to your container registry 3. Helm setup: - Helm is installed and configured - The "jetstack" repository is added: helm repo add jetstack https://charts.jetstack.io - The cert-manager and trust-manager charts are installed via helm Also review the 'Prerequisites for Cloud Scale Technology upgrade' section in the 'NetBackup™ Deployment Guide for Kubernetes Clusters' document for additional required steps. Once everything is ready, you can safely continue with the upgrade. {Component: CloudScale, ComponentName: Upgrade of CloudScale} INFO: 2025/09/26 12:38:50 logger.go:132: Would you like to continue? (y/n): {Component: CloudScale, ComponentName: Upgrade of CloudScale} INFO: 2025/09/26 12:39:00 logger.go:132: Helm Version: version.BuildInfo{Version:"v3.18.6", GitCommit:"b76a950f6835474e0906b96c9ec68a2eff3a6430", GitTreeState:"clean", GoVersion:"go1.24.6"} {Component: Precheck Config, ComponentName: Installation of CloudScale} INFO: 2025/09/26 12:39:00 logger.go:132: kubectl Version: Client Version: v1.34.1 Kustomize Version: v5.7.1 Server Version: v1.32.6 {Component: Precheck Config, ComponentName: Installation of CloudScale} INFO: 2025/09/26 12:39:00 logger.go:132: ************************************************** {Component: CloudScale Upgrade, ComponentName: Input Configuration} INFO: 2025/09/26 12:39:00 logger.go:129: Checking if the input file already exists. INFO: 2025/09/26 12:39:00 logger.go:132: Data is being read from an input file that is present. {Component: CloudScale, ComponentName: Input Configuration} INFO: 2025/09/26 12:39:00 logger.go:132: The following values were loaded from the file: {Component: CloudScale Upgrade, ComponentName: Input Configuration}
The plugin was cancelled while checking for Cert Manager installation. When the upgrade was triggered again, it skipped the Cert Manager validation step and proceeded to the next question instead of restarting the validation process.
Workaround:
Delete the following file and re-run the upgrade:
/home/<user-name>/kubectl-plugin/upgrade.csv
The plugin was cancelled before completing the operator upgrade. When the upgrade was triggered again, it failed due to the Helm release being stuck in a pending-upgrade state with the following error:
Operator Namespace : netbackup-operator-system Upgrade of operators started... Helm upgrade of operators failed with error: another operation (install/upgrade/rollback) is in progress Operators chart failed to upgrade. Error while upgrading operators chart : another operation (install/upgrade/rollback) is in progress Upgrade failed:
Workaround:
To resolve this issue:
Check for pending Helm releases:
helm ls -A --pending
If the operator's release is in a pending-upgrade state, list previous revisions:
helm history operators --namespace <operator_namespace>
Identify a revision with the status or and roll back to that version:
helm rollback operators <REVISION> --namespace <operator_namespace>
This clears the pending-upgrade lock caused by the interrupted upgrade.
Once the rollback completes, rerun the plugin upgrade command:
kubectl-cloudscale upgrade
When prompted to resume the installation, type to continue from where it stopped.
The plugin crashed while annotating the environment resources immediately after the operator upgrade.
Workaround:
Trigger the upgrade process again using the
kubectl-cloudscaleplugin:kubectl-cloudscale upgrade
When prompted to resume the installation, type to continue the upgrade from where it stopped.
If the plugin crashes at any point before Helm triggers the Cloud Scale upgrade command, the upgrade process is interrupted and cannot complete successfully.
Workaround:
Trigger the upgrade process using the
kubectl-cloudscaleplugin:kubectl-cloudscale upgrade
When prompted to resume the installation, type to continue the upgrade from where it stopped.
The media server pod is continuously restarting. Upon inspection, it is observed that the mount point /mnt/nblogs inside the media pod is in a read-only state, which is likely causing the restarts.
Perform the following steps to verify if the mount point is read-only
- Exec the following command into the media pod:
kubectl exec -it <media-pod-name> -- bash
- Run the following command to check the mount point permissions:
cd /mnt/nblogs/fluentbit
touch test
Expected output (in case of an issue):
touch: cannot touch 'test': Read-only file system
The above error message confirms that the
/mnt/nblogsmount is mounted as read-only.
Workaround:
Run the following command to manually restart the media pod by deleting the affected media pod:
kubectl delete pod <media-pod-name>