Post Kubernetes cluster restart, issues observed in case of containerized Postgres deployment
After Kubernetes cluster restart in case of containerized Postgres deployment, couple of flexsnap pods went into state and primary pod was not started successfully.
This issue arises due to certificate expiration when the cluster was down. Perform the following to ensure that the issue occurred was due to certificate expiration:
Run the following command to exec into the
nb-postgresqlpod:kubectl -n <namespace> exec pod/nb-postgresql-0 -it bash
Run the following command from within the Postgres pod
psql "host=nb-postgresql port=13787 dbname=postgres user=postgres sslmode=verify-full sslcert='/netbackup/postgresql/keys/server/tls.crt' sslkey='/netbackup/postgresql/keys/server/tls.key' sslrootcert=/netbackup/postgresql/keys/server/ca.crt"
Following error message is displayed:
psql: error: connection to server at "nb-postgresql" (10.240.3.27), port 13787 failed: SSL error: certificate verify failed.
Run the following command to check the
ca.crtserver certificate:Get the
ca.crtof :kubectl -n <namespace> get secret postgresql-server-crt -o "jsonpath={.data['ca\.crt']}" | base64 -d > server_ca.crt
Get the
ca.crtof :kubectl -n <namespace> get secret postgresql-netbackup-ca -o "jsonpath={.data['ca\.crt']}" | base64 -d > ca.crt
Get current date on the system:
date
Following message is displayed:
Tue Feb 27 17:07:14 UTC 2024
Check expiry of
ca.crt:openssl x509 -enddate -noout -in ca.crt
Following message is displayed:
notAfter=Feb 27 17:52:58 2024 GMT
Check expiry of
server_ca.crt:openssl x509 -enddate -noout -in server_ca.crt
Following message is displayed:
notAfter=Feb 27 16:57:58 2024 GMT
The above displayed messages indicate that
ca.crtof is referring to an expired certificate.
Resolution:
Perform the following to resolve the certificate issue:
Delete the certificate secrets:
kubectl -n <namespace> delete secret postgresql-netbackup-ca postgresql-server-crt postgresql-client-crt
Execute the following commands to ensure that the correct certificate is referred:
Get the
ca.crtof :kubectl -n <namespace> get secret postgresql-server-crt -o "jsonpath={.data['ca\.crt']}" | base64 -d > server_ca.crt
Get the
ca.crtof :kubectl -n <namespace> get secret postgresql-netbackup-ca -o "jsonpath={.data['ca\.crt']}" | base64 -d > ca.crt
Get current date on the system:
date
Following message is displayed:
Tue Feb 27 17:11:59 UTC 2024
Check expiry of
ca.crt:openssl x509 -enddate -noout -in ca.crt
Following message is displayed:
notAfter=Feb 27 18:11:38 2024 GMT
Check expiry of
server_ca.crt:openssl x509 -enddate -noout -in server_ca.crt
Following message is displayed:
notAfter=Feb 27 18:11:38 2024 GMT
The above displayed messages indicate that
ca.crtof is referring to the correct certificate.Run the following command to exec into the
nb-postgresqlpod:kubectl -n <namespace> exec pod/nb-postgresql-0 -it bash
Run the following command from within the Postgres pod
psql "host=nb-postgresql port=13787 dbname=postgres user=postgres sslmode=verify-full sslcert='/netbackup/postgresql/keys/server/tls.crt' sslkey='/netbackup/postgresql/keys/server/tls.key' sslrootcert=/netbackup/postgresql/keys/server/ca.crt"
Following message is displayed:
Password for user postgres: