NetBackup status code: 41
Message: network connection timed out
Explanation: The possible causes include the following:
The server did not receive any information from the client for too long a period of time.
You have attempted to back up too many files using FSCP (File level fast single copy backup) with fast backup enabled.
NetBackup uses the available bandwidth to its maximum potential and pushes the requests accordingly; however, the Amazon S3 compliant cloud cannot process the high number of requests.
The cloud vendor returns error 503 to slow down the requests and the backup job fails with the following errors:
In the media server, bptm logs the following:
bptm:4940:<media_server_name>: AmzResiliency: AmzResiliency::getRetryType cURL error: 0, multi cURL error: 0, HTTP status: 503, XML response: SlowDown, RetryType: RETRY_EXHAUSTED
In the media server, bpbrm logs the following:
bpbrm Exit: client backup EXIT STATUS 41: network connection timed out
This issue occurs only if a higher bandwidth is available between NetBackup and the cloud storage.
Connection between the master server and a NAT client failed.
Recommended Action: Do the following, as appropriate:
If you backup to many files, use Host Properties on the NetBackup server to change to a much higher value (for example: 4000). These settings are on the Universal Settings tab in the Master Server Properties dialog box. The default for this time out is 300 seconds.
Also, set the File browse timeouton the Timeouts tab to a value greater than 4000.
Then retry the operation. The names of the files are logged on the debug log file in the
/usr/openv/netbackup/logs/bpbkardirectory before bpbkar processes them. The last file in the log is the file that causes problems.On UNIX, Linux, or Windows clients, check for the following problems with the bpbkar client process.
On Windows clients: The bpbkar client process is not hung. Due to the files and directories it scans, it has not replied to the server within the period. This error occurs during incremental backups when directories have thousands of unmodified files.
For this case, use Host Properties on the NetBackup server to change . This setting is on the Universal Settings tab in the Master Server Properties dialog box. The default for this time out is 300 seconds.
See "Using the Host Properties window to access configuration settings" in the NetBackup Troubleshooting Guide.
You can also monitor CPU utilization to determine if this condition exists.
The following information applies only to UNIX or Linux clients:
The bpbkar client process is hung on a file that has a mandatory locking set. For this case, add the following to the client's
bp.conffile:VERBOSE
As root on the client, run the following:
touch /usr/openv/netbackup/bpbkar_path_tr /usr/openv/netbackup/logs/bpbkar
Then retry the operation. The names of the files are logged on the debug log file in the
/usr/openv/netbackup/logs/bpbkardirectory before bpbkar processes them. The last file in the log is the file that causes problems.Note:
Also, use these procedures for other unknown bpbkar hangs.
If the problem is due to mandatory file locking, have NetBackup skip the locked files. Set LOCKED_FILE_ACTION to SKIP in the
/usr/openv/netbackup/bp.conffile on the client.The bpbkar client process is not hung. Due to the files and directories it scans, it has not replied to the server within CLIENT_READ_TIMEOUT or CLIENT_CONNECT_TIMEOUT. This error occurs during backups when directories have thousands of unmodified files or during restores of the sparse files that have thousands of holes. For this case, try to add or modify the CLIENT_READ_TIMEOUT value in the server's /usr/openv/netbackup/bp.conf file. The default for CLIENT_READ_TIMEOUT is 300 seconds if it is not specified.
Use your system's ps command and monitor CPU utilization to help decide which of these conditions exist.
When you finish the investigation of the problem, delete the
/usr/openv/netbackup/logs/bpbkardirectory, since the log files can become quite large and are not deleted automatically. Also delete/usr/openv/netbackup/bpbkar_path_trso you do not generate larger log files than needed the next time you create directory/usr/openv/netbackup/logs/bpbkar.On Windows systems, try the following:
Disable the following file:
install_path\VERITAS\NetBackup\bin\tracker.exe
Repair hard drive fragmentation. Try an application that is called Diskeeper Lite, which is part of the Windows Resource Kit.
Make sure that enough space is available in
\temp.
If the server cannot connect to the client, create bpcd or bpbkar (UNIX, Linux, and Windows only) debug log directories on the client. Then retry the operation and check the resulting logs. If these logs do not provide a clue, create a bpbrm debug log on the server. Then retry the operation and check the resulting debug log.
If the bpbrm log has entries similar to the following, the problem is in the routing configuration on the server:
bpbrm hookup_timeout: timed out waiting during the client hookup bpbrm Exit: client backup EXIT STATUS 41: network connection timed out
Verify that the client IP address is correct in the name service that is used. On UNIX clients, if both the NIS and the DNS files are used, verify that they match.
See "Resolving network communication problems" in the NetBackup Troubleshooting Guide.
If you use an AIX token ring adapter and the routed daemon is running, the time out occurs because the token ring adapter creates dynamic routes. It then causes the routed daemon to crash.
For a FlashBackup client, this error occurs if the file system being backed up is very large and has a very large number of files. It can also occur if a large number of concurrent data streams are active at the same time. To correct it, add CLIENT_READ_TIMEOUT to the
/usr/openv/netbackup/bp.conffile and set it to increase the time-out interval.Check the Cohesity Technical Support website to ensure that all recommended NetBackup patches are installed.
Add the CLIENT_READ_TIMEOUT values to the master server, media server, and client when a NetBackup database extension product is installed. The values should all be the same for each server. The value set is dependent on the size of the database being backed up. More information on CLIENT_READ_TIMEOUT is available.
Make sure that enhanced authentication is configured correctly. For example, the following may result in status code 41: Host A is configured to use enhanced authentication with host B, but host B is not configured to use enhanced authentication with host A. In this case, connections from host B to host A are likely to fail with status code 41. Connections from host A to B are likely to fail with authentication errors (status code 160).
If the Amazon S3 compliant cloud is not able to process the high number of requests, perform one of the following:
Configure the bandwidth throttling to reduce the number of requests. See "NetBackup cloud storage server connection properties" in the NetBackup Cloud Administrator's Guide.
Reduce the number of read/write buffers. See "NetBackup cloud storage server bandwidth throttling properties" in the NetBackup Cloud Administrator's Guide.
Ask your cloud vendor to increase the parallel requests limit.
If connection between the master server and a NAT client failed, do the following:
Ensure that the subscriber service on the client is up and running.
If the problem persists, restart the client services and ensure that the subscriber service is connected with the NetBackup Messaging Broker (or
nbmqbroker) service.
Click here to view technical notes and other information on the Cohesity Technical Support website about this status code.