Cloud storage backups fail
See the following topics:
A message similar to the following is in the job details:
Critical bptm(pid=28291) accelerator verification failed: backupid= host_name_1373526632, offset=3584, length=141976576, error= 2060022, error message: software error Critical bptm(pid=28291) image write failed: error 2060022: software error Error bptm(pid=28291) cannot write image to disk, Invalid argument end writing; write time: 0:02:31 Info bptm(pid=28291) EXITING with status 84 Info bpbkar(pid=6044) done. status: 84: media write error media write error(84)
This error may occur in the environments that have more than one cloud storage server. It indicates that NetBackup Accelerator backups of a client to one cloud storage server were later directed to a different cloud storage server.
For Accelerator backups to cloud storage, ensure the following:
Always back up each client to the same storage server. Do so even if the other storage server represents storage from the same cloud storage vendor.
Always use the same backup policy to back up a client, and do not change the storage destination of that policy.
If the cloud storage server WRITE_BUFFER_SIZE property exceeds the total swap space of the computer, backups can fail with a status 84.
Adjust the WRITE_BUFFER_SIZE size to a value lower than the computer's total swap space to resolve this issue.
A message similar to the following is in the job details:
Info bptm(pid=xxx) start backup Critical bptm(pid=xxxx) image open failed: error 2060029: authorization failure Error bpbrm(pid=xxxx) from client gabby: ERR - Cannot write to STDOUT. Errno = 32: Broken pipe Info bptm(pid=xxxx) EXITING with status 84
A message similar to the following appears in the bptm log file:
Container container_name is not Cohesity container or tag data error, fail to create image. Please make sure that the LSU is created by means of NBU.
This error indicates that the volume was created by using the cloud storage vendor's interface.
Use the NetBackup web UI to create the volume on the cloud storage. NetBackup applies a required partner ID to the volume. If you use the vendor interface to create the container, the partner ID is not applied.
To resolve the problem, use the cloud storage vendor's interface to delete the container. In NetBackup, delete the disk pool and then recreate it with web UI.
See Viewing cloud storage job details.
See NetBackup cloud storage log files.
This is applicable to media server versions 7.7.x to 8.1.2 only.
If the NetBackup CloudStore Service Container is not active, backups cannot be sent to the cloud storage.
NetBackup does not validate that the CloudStore Service Container is active when you use NetBackup commands to configure NetBackup cloud storage. Therefore, any backups that initiate in such a scenario fail.
See NetBackup CloudStore Service Container startup and shutdown troubleshooting.
While you configure a cloud storage server, you must ensure that the media server and the primary server are of the same version.
Note:
This limitation does not apply to the existing cloud storage servers.
Cloud backups may fail in the following scenario:
You selected while you configured the storage unit and NetBackup uses a media server with version different than the primary server version during cloud storage configuration.
To resolve this issue, do the following:
Select while you configure the storage unit and select the media server with a version same as primary server from the Media Servers pane.
The cloud backups and restore operations failing with error code 83 or error code 2106 may occur due to any one of the following reasons:
The media server's date and time settings are skewed (not in sync with the GMT/UTC time).
The storage server credentials that are provided are incorrect.
Perform the following:
Change the media server's date and time settings so that it is in sync with the GMT/UTC time.
Update the storage server credentials. Use the tpconfig command to update the credentials. For more information, see the NetBackup Commands Reference Guide.
If the cloud storage backups fails because of certificate issues, verify the following:
The NetBackup
cacert.pemfile is present on both NetBackup primary and media server in following locations:UNIX/Linux -
/usr/openv/var/webtruststoreWindows -
<install_path>/var/webtruststore
For media server versions 7.7.x to 8.1.2, if the NetBackup
cacert.pemfile is not present, run the nbcertcmd -getCACertificate on the primary server. After running this command, restart the NetBackup CloudStore Service Container.See the NetBackup Commands Reference Guide for a complete description of the command.
Note:
This NetBackup
cacert.pemfile is a NetBackup-specific file. This file includes the CA certificates generated by the NetBackup authorization service.The NetBackup
cacert.pemfile is same on the NetBackup primary and media server.For media server versions 7.7.x to 8.1.2, the machine certificate is present in following locations:
UNIX/Linux -
/usr/openv/var/vxss/credentialsWindows -
<install_path>/var/vxss/credentials
If the security certificate is not present, run the bpnbaz -ProvisionCert on the primary server. After running this command, restart the NetBackup CloudStore Service Container on the primary and media server.
For media server versions 7.7.x to 8.1.2, the NetBackup CloudStore Service Container is active.
See Stopping and starting the NetBackup CloudStore Service Container.
The option on the NetBackup primary server is selected if the media server is of the version 8.0 or earlier. The option is available in the NetBackup Administration Console on the Security Management > Global Security Settings > Secure Communication tab.
On the media server, if the certificate deployment security level if set to Very High, automatic certificate deployment is disabled. An authorization token must accompany every new certificate request. Therefore, you must create an authorization token before deploying the certificates.
See the 'Creating authorization tokens' topic in the NetBackup™ Security and Encryption Guide for detailed steps.
NetBackup consumes the available bandwidth to it's maximum potential and pushes the requests accordingly, however the Amazon S3 complaint cloud is not able to process the number requests.
The cloud vendor returns error 503 to slow down the requests and the backup job fails with the following errors:
In the media server
bptmlogs:bptm:4940:<media_server_name>: AmzResiliency: AmzResiliency::getRetryType cURL error: 0, multi cURL error: 0, HTTP status: 503, XML response: SlowDown, RetryType: RETRY_EXHAUSTED
In the media server
bpbrmlogs:bpbrm Exit: client backup EXIT STATUS 41: network connection timed out
This issue arises only if higher bandwidth is available between NetBackup and the cloud storage.
To troubleshoot you can perform one of the following:
Configure bandwidth throttling to reduce the number of requests.
Reduce the number of read/write buffers.
See NetBackup cloud storage server bandwidth throttling properties.
Talk to your cloud vendor to increase the number of parallel requests limit. This might incur extra cost.
More Information