Ensure that the container is running the patched image
There are three copies of the container image present in the Kubernetes environment during deployment or patching.
The first image copy is created on a local docker instance during image load operation. To check this copy, perform the following:
- Run:
$ docker load -i images/pdk8soptr-20.5.tar.gz
Sample output:
Loaded image: msdp-operator:20.5 - Taking the image name from step 1, run:
$ docker image ls | grep msdp-operator
Sample output:
msdp-operator 20.5 353d2bd50105 2 days ago 480 MB - Taking the value from step 2, run:
$ docker inspect 353d2bd50105 | jq .[].Id
"sha256:353d2bd50105cbc3c61540e10cf32a152432d5173bb6318b8e"
The second copy is created in the following respective registry's:
(AKS-specific): Azure Container Registry (ACR)
(EKS-specific): Amazon Elastic Container Registry (ECR)
To check this copy, perform the following:
- Keep the image name and version same as original, run:
(AKS-specific): $ docker image tag msdp-operator:20.5 testregistry.azurecr.io/msdp-operator:20.5
(EKS-specific): $ docker image tag msdp-operator:20.5 046777922665.dkr.ecr.us-east-2.amazonaws.com/nbuxeksdeploy.aws.io/msdp-operator:20.5
- Run:
$ docker image ls | grep msdp-operator
Sample output:
(AKS-specific):
msdp-operator 20.5 353d2bd50105 2 days ago 480 MB tregistry.azurecr.io/msdp-operator 20.5 353d2bd50105 2 days ago 480 MB
(EKS-specific):
msdp-operator 20.5 353d2bd50105 2 days ago 480 MB msdp-operator 20.5 046777922665.dkr.ecr.us-east-2. amazonaws.com/nbuxeksdeploy.aws.io/msdp-operator 20.5 353d2bd50105 2 days ago 480 MB /msdp-operator 20.5 353d2bd50105 2 days ago 480 MB
- To push the image to the registry, run:
(AKS-specific): $ docker push testregistry.azurecr.io/msdp-operator
(EKS-specific): $ docker push testregistry.<account id>.dkr.ecr.<region>.amazonaws.com/<registry>:<tag>.io/msdp-operator
The push refers to a repository [testregistry.azurecr.io/msdp-operator]
(AKS-specific): [testregistry.azurecr.io/msdp-operator]
(EKS-specific): [046777922665.dkr.ecr.us-east-2.amazonaws.com/nbuxeksdeploy.aws.io/msdp-operator]
0a504041c925: Layer already exists
20.5: digest: sha256:d294f260813599562eb5ace9e0acd91d61b7dbc53c3 size: 2622
- To verify local image digest after the push operation, run:
$ docker inspect 353d2bd50105 | jq .[].RepoDigests
Sample output:
(AKS-specific):
[ "testregistry.azurecr.io/msdp-operator@sha256: d294f260813599562eb5ace9e0acd91d61b7dbc53c3" ]
(EKS-specific):
[ "testregistry.<account id>.dkr.ecr.<region>.amazonaws.com/<registry>:<tag>.io/ msdp-operator@sha256: d294f260813599562eb5ace9e0acd91d61b7dbc53c3" ]
- To verify image presence in the registry, run:
(AKS-specific): $ az acr repository list --name testregistry
(EKS-specific): $ aws ecr describe-repositories --repository-names "veritas/main_test1"
Sample output:
(AKS-specific):
[ "msdp-operator", ]
(EKS-specific):
"repositories": [ { "repositoryArn": "arn:aws:ecr:us-east-2:046777922665:repository/veritas/main_test1", "registryId": "046777922665", "repositoryName": "veritas/main_test1", "repositoryUri": "046777922665.dkr.ecr.us-east-2.amazonaws.com/veritas/main_test1", "createdAt": "2022-04-13T07:27:52+00:00", "imageTagMutability": "MUTABLE", "imageScanningConfiguration": { "scanOnPush": false }, "encryptionConfiguration": { "encryptionType": "AES256" } } ] - To verify image digest in registry, run:
(AKS-specific): $ az acr repository show -n testregistry --image msdp-operator:20.5
(EKS-specific): $ aws ecr describe-images --registry-id 046777922665 --repository-name "veritas/main_test1" --image-ids imageTag=latestTest5
Sample output:
(AKS-specific):
{ "changeableAttributes": { "deleteEnabled": true, "listEnabled": true, "readEnabled": true, "writeEnabled": true }, "createdTime": "2022-02-01T13:43:26.6809388Z", "digest": "sha256:d294f260813599562eb5ace9e0acd91d61b7dbc53c3", "lastUpdateTime": "2022-02-01T13:43:26.6809388Z", "name": "20.5", "signed": false }(EKS-specific):
"imageDetails": [ { "registryId": "046777922665", "repositoryName": "veritas/main_test1", "imageDigest": "sha256:d0095074286a50c6bca3daeddbaf264cf4006a92fa3a074daa4739cc995b36f8", "imageTags": [ "latestTest5" ], "imageSizeInBytes": 38995046, "imagePushedAt": "2022-04-13T15:56:07+00:00", "imageManifestMediaType": "application/vnd.docker. distribution.manifest.v2+json", "artifactMediaType": "application/vnd.docker.container.image.v1+json" } ]
The third copy is located on a Kubernetes node running the container after it is pulled from the registry. To check this copy, perform the following:
- Run;
$ kubectl get nodes -o wide
(AKS-specific):
NAME STATUS VERSION INTERNAL-IP OS-IMAGE aks-agentpool-7601-vmss000 Ready v1.21.7 10.240.0.4 Ubuntu 20.54.6 LTS
(EKS-specific):
NAME STATUS VERSION INTERNAL-IP OS-IMAGE eks-agentpool-7601-vmss000 Ready v1.21.7 10.240.0.4 Ubuntu 20.54.6 LTS
- Use kubectl debug to run a container on the node:
(AKS-specific): $ kubectl debug node/aks-nodepool1-7601-vmss000-it --image=mcr.microsoft.com/aks/fundamental/base-ubuntu:v0.0.11 root@aks-agentpool-7601-vmss000:/#
(EKS-specific): $ kubectl debug node/eks-nodepool1-7601-vmss000-it --image=mcr.microsoft.com/eks/fundamental/base-ubuntu:v0.0.11 root@eks-agentpool-7601-vmss000:/#
- You can interact with the node session from the privileged container:
chroot /host
- Verify the presence of the image:
/usr/local/bin/crictl image | grep msdp
Sample output:
(AKS-specific):
testregistry.azurecr.io/msdp-operator 20.5 353d2bd50105c 182MB
(EKS-specific):
<account id>.dkr.ecr.<region>.amazonaws.com/msdp-operator 20.5 353d2bd50105c 182MB
- Verify the image ID on the Kubernetes node, run:
/usr/local/bin/crictl inspecti 353d2bd50105c | jq .[].id
Sample output
"sha256:353d2bd50105cbc3c61540e10cf32a152432d5173bb6318b8e" null
- Verify the image digest on the Kubernetes node, run:
/usr/local/bin/crictl inspecti 353d2bd50105c | jq .[].repoDigests
Sample output
(AKS-specific):
[ "testregistry.azurecr.io/msdp-operator@sha256: d294f260813599562eb5ace9e0acd91d61b7dbc53c3" ] null
(EKS-specific):
[ "<account id>.dkr.ecr.<region>.amazonaws.com/msdp-operator@sha256: d294f260813599562eb5ace9e0acd91d61b7dbc53c3" ] null
Use the steps given above to identify image ID and Digest and compare with values obtained from the registry and the Kubernetes node running the container.
Note:
MSDP Scaleout images (uss-engine, uss-mds, uss-controller, msdp-operator) use IfNotPresent imagePullPolicy. A unique image tag is required in order for a Kubernetes node to pull an updated image.