Troubleshoot PaaS workload protection and recovery issues
You can see the following message in the Activity Monitor:
AuthorizationFailed -Message: The client '<clientId> '<objetId>' does not have authorization to perform action 'Microsoft.Sql/servers/databases/read' over scope '<resoourceId>' or the scope is invalid. If access was recently granted, please refresh your credentials.
Explanation: This error occurs, when the Snapshot Manager and NetBackup are deployed in AKS, and:
The media server pod node pool is a different node pool from the Snapshot Manager node pool
Managed Identity is enabled in the Snapshot Manager Virtual Machine Scale set
Workaround: Do any of the following:
In the media server that you are using for backup and restore, enable Managed Identity on the Scale set. Also, assign required permission in the role attached to this managed identity.
Create a storage unit on the MSDP server, and use only those media servers that has Managed Identity enabled on Scale configuration.
Explanation: This issue occurs if the Read-only lock or Delete lock attribute is applied to the database or the resource group.
Workaround: Before performing any backup or restore, remove any existing Read-only lock and Delete lock attributes from the database or the resource group.
Explanation: This appears when you manually cancel a backup or a restore job from the activity monitor and a database is created on the portal during the partial restore operation.
Workaround:Manually clean up the database on the provider portal, and temporary staging location at the universal share mount location under specific directory created by database name.
Explanation: If the Snapshot Manager container service restarts abruptly; the provider protected restore jobs may remain in the active state and you may not see the updated status on the activity monitor details page.
Workaround: Restart the workflow containers using the following command in the Snapshot Manager:
docker restart flexsnap-workflow-system-0-min flexsnap-workflow-general-0-min
After restarting the containers, the restore jobs are updated with the latest status in the activity monitor.
Explanation: Appears if the client name used for backup exceeds the length of 255 characters.
The bpdbm logs confirms the same by displaying the following error message:
db_error_add_to_file: Length of client is too long. Got 278, but limit is 255. read_next_image: db_IMAGEreceive() failed: text exceeded allowed length (225)
Note:
This is observed when the primary server is RHEL.
Workaround:Rename the database such that the client name fits within the length of 255 characters.
Explanation:Occurs during backup if the client name used is big in length, due to this the file path length of catalog image exceeds 256 chars and fails with the above error message in activity monitor.
The bpdbm logs confirms the same by displaying the following error message:
<16> db_error_add_to_file: cannot stat(\\?\C:\Program Files\Veritas \NetBackup\db\images \azure-midb-1afb87487dc04ddc8fafe453dccb7ca3+ nbux-qa-bidi-rg+eastus+az-sql-mi-bidinet01+ testdb_bidinet02\1656000000\tmp\catstore\ BACKUPNOW+141a73e7-cdc4-4371-823a-f170447dba2d_ 1656349831_FULL.f_imgUserGroupNames0): No such file or directory (2) <16> ImageReadFilesFile::get_file_size: cannot stat(\\?\C:\Program Files\Veritas\NetBackup\db \images\azure-midb-1afb87487dc04ddc8fafe453d ccb7ca3+nbux-qa-bidi-rg+eastus+az-sql-mi-bidinet01+testdb_ bidinet02\1656000000\tmp\catstore\BACKUPNOW+141a73e7-cdc4-4371 -823a-f170447dba2d_1656349831_FULL.f_imgUserGroupNames0): No such file or directory (2) <16> ImageReadFilesFile::executeQuery: Cannot copy \\?\C:\Program Files\Veritas\NetBackup\db\images\azure-midb-1afb87487dc04ddc8fafe453dccb7 ca3+nbux-qa-bidi-rg+eastus+az-sql-mi-bidinet01+testdb_bidinet02\1 656000000\tmp\catstore\BACKUPNOW+141a73e7-cdc4-4371-823a-f170447d ba2d_1656349831_FULL.f_imgUserGroupNames0
Note:
This is observed when the primary server is Windows.
Workaround: Rename the database such that the file path length fits within the length of 256 characters.
Explanation:NetBackup is not able to successfully carry out the requested operation.
Recommended action: Please refer to the activity monitor details for possible reason of failure.
Explanation: The error message seen in dbagentsutil logs as,pg_dump: error: query failed: ERROR: permission denied for table test;pg_dump: error: query was: LOCK TABLE public.test IN ACCESS SHARE MODE;Invoked operation: PRE_BACKUP failed
Occurs when you try to backup a database which has multiple tables with different roles. If tables have at least one different owner, other than the database owner, and it is not a member the database owner role, then the backup may fail.
Recommended action: You must have a role that has access to all tables inside the database which you want to backup or restore.
For example, say that we wanted to backup the School database which has two tables.
student,owner ispostgresteacher,owner isschooladmin
Create a new role. Say, NBUbackupadmin
Run the following command to create the role:
postgres=> CREATE USER NBUbackupadmin WITH PASSWORD '***********';
CREATE ROLE
To make this new role a member of postgres and schooladmin role, run:
postgres=> GRANT postgres TO NBUbackupadmin;
GRANT ROLE
postgres=> GRANT schooladmin TO NBUbackupadmin;
GRANT ROLE
Note:
You must have a role who is either owner or member of the owner of the table, for all tables inside database.
Explanation: Backups fail due to loss of connectivity to the media server.
Recommended action: You can restart the backup job if the policy has checkpoints enabled. Once the network issue is resolved, select the incomplete backup job in web UI and click . The job resumes from the point it was stopped. If checkpoint is not enabled in the policy, the job shows up as a failed job in the web UI.