Acknowledgment not received for datamover
Backup job fails with the following error message, where acknowledgment is not received for datamover:
Oct 10, 2022 5:06:21 AM - begin SnapDupe Mount: Import Snapshot Oct 10, 2022 5:06:21 AM - Info nbjm (pid=7578) BackupId=aws-ec2-us-east-2-xxxxxxxxxxxxxx_1665303611 Oct 10, 2022 5:06:23 AM - Info nbcs (pid=523) Start Oct 10, 2022 5:06:23 AM - Info nbcs (pid=523) Requesting data mover container Oct 10, 2022 5:18:36 AM - Error nbcs (pid=523) Invalid operation for asset: aws-ec2-us-east-2-xxxxxxxxxxxxxx Oct 10, 2022 5:18:36 AM - Error nbcs (pid=523) Acknowledgment not received for datamover datamover.a2d3dc2249da45a0a839bc77eface2a4 Oct 10, 2022 5:18:36 AM - Info nbcs (pid=523) End
The above error message is observed on the cluster, when:
The pods are in ContainerCreating state. For example:
flexsnap-workflow-general-1665398188-4d03f27e-fblxb 0/1 ContainerCreating 0 142m flexsnap-workflow-general-1665398188-538a8846-zrgtl 0/1 ContainerCreating 0 142m flexsnap-workflow-general-1665398188-87cb301a-5bqss 0/1 ContainerCreating 0 142m flexsnap-workflow-general-1665398188-f61f5f42-g2rhv 0/1 ContainerCreating 0 142mThe describe pod displays the events as follows:
Type Reason Age From Message ---- ------ ---- ---- ------- Normal SandboxChanged 25m (x1874 over 140m) kubelet Pod sandbox changed, it will be killed and re-created. Warning FailedCreatePodSandBox 56s (x2079 over 140m) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc =failed to set up sandbox container "45f90b441cc4ea83efca63eacff1028779d4114fb213a5200e76f2e25373e054" network for pod "flexsnap-workflow-general-1665398189-f46e636e-vrcdz": networkPlugin cni failed to set up pod "flexsnap-workflow-general-1665398189-f46e636e-vrcdz_nbuxsystest" network: add cmd: failed to assign an IP address to container
To resolve this issue, refer to the AWS troubleshooting section and implement the solution. Contact the AWS support for further troubleshooting.