Socket connection failure
Socket connection failure can happen because of the following reasons:
Long processing delays
Azure/AWS connection reset (default 4 minutes)
Load on CPU or Memory pressure
IO saturation and throttling under load
If there are problems with the TCP stacks on the hosts, network between the hosts, or unusual long processing delays, then the connection may drop and the TCP stack on the host is unaware of the situation.
The following error is displayed in the web UI under job details:
db_FLISTsend failed: unexpected message received (43) *** - Error bptm (pid=14599) get_string() failed, Connection reset by peer (104), network read error *** - Info bptm (pid=14599) EXITING with status 42 <---------- *** - Info nbux-systest-media-1 (pid=14599) StorageServer=PureDisk:nbux-systest-media-1; Report=PDDO Stats for (nbux-systest-media-1): scanned: 4195521 KB, CR sent: 171002 KB, CR sent over FC: 0 KB, dedup: 95.9%, cache disabled, where dedup space saving:6.6%, compression space saving:89.3% *** - Info bpbkar (pid=19109) done. status: 42: network read failed
To resolve this issue, update the sysctl.conf values for NetBackup servers deployed on the Kubernetes cluster.
NetBackup image sets following values in sysctl.conf during Kubernetes deployment:
net.ipv4.tcp_keepalive_time = 180
net.ipv4.tcp_keepalive_intvl = 10
net.ipv4.tcp_keepalive_probes = 20
net.ipv4.ip_local_port_range = 14000 65535
These settings are persisted at the location /mnt/nbdata/etc/sysctl.conf.
Modify the values in /mnt/nbdata/etc/sysctl.conf and restart the pod. The new values are reflected after the pod restart.
If external media servers are used, perform the steps in the following order:
Add the following in
/usr/openv/netbackup/bp.conf:Add the following sysctl configuration values in
etc/sysctl.confon external media servers to avoid any socket connection issues:net.ipv4.tcp_keepalive_time = 180
net.ipv4.tcp_keepalive_intvl = 10
net.ipv4.tcp_keepalive_probes = 20
net.ipv4.ip_local_port_range = 14000 65535
net.core.somaxconn = 4096
Save the setting using the sysctl -p command.