Azure-specific

1. Launch an Azure CLI pod into the AKS cluster using the following command:

   $ kubectl run az-cli --image=mcr.microsoft.com/azure-cli:2.53.0 --command sleep infinity

   Note:

   Access to Azure Key Vault is restricted to specific subnets. Passwords stored in Azure Key Vault can be easily updated from a pod running in AKS.

   Connecting to Postgres database using Azure requires installing rdbms-connections. This functionality is applicable to azure-cli 2.53.0.

2. Exec into the Azure CLI pod as follows:

   $ kubectl exec -it az-cli -- /bin/ash

3. From Azure CLI pod, log into Azure account:

   $ az login --scope https://graph.microsoft.com//.default

4. (Optional) Create a key vault policy to allow the current user to retrieve the database credential.

   Obtain the name of your resource group, key vault and ID of the current user by using the following respective commands:

   • Resource group name:

     $ RESOURCE_GROUP=<resource_group_name>

   • Key vault name:

     $ KEY_VAULT_NAME=$(az keyvault list --resource-group $RESOURCE_GROUP --resource-type vault | jq -r '.[].name')

   • Current user ID name:

     $ USER_ID=$(az account show | jq -r '.user.name')

   Create a key vault access policy as follows:

   $ az keyvault set-policy -n $KEY_VAULT_NAME --upn $USER_ID --resource-group $RESOURCE_GROUP --secret-permissions all

5. Obtain the login name for the key vault (DBADMINUSER):

   $ DBADMINUSER=$(az keyvault secret show --vault-name $KEY_VAULT_NAME --name dbadminlogin | jq -r .value)

6. Obtain the password for the key vault (OLD_DBADMINPASSWORD):

   $ OLD_DBADMINPASSWORD=$(az keyvault secret show --vault-name $KEY_VAULT_NAME --name dbadminpassword | jq -r .value)

7. Obtain the server name (DBSERVER):

   DBSERVER=$(az postgres flexible-server list --resource-group $RESOURCE_GROUP | jq -r '.[].name')

8. (Optional) Verify the current password encryption method by using the following command:

   az postgres flexible-server execute -p "$OLD_DBADMINPASSWORD" -u $DBADMINUSER -n $DBSERVER -d postgres -q "SELECT * from azure_roles_authtype();" -o table

   Following message is displayed:

   Successfully connected to twilk-db.
   Ran Database Query: 'SELECT * from azure_roles_authtype();'
   Retrieving first 30 rows of query output, if applicable.
   Closed the connection to twilk-db
   Authtype    Rolename
   ----------  -------------------------
   NOLOGIN     azuresu
   NOLOGIN     pg_database_owner
   NOLOGIN     pg_read_all_data
   NOLOGIN     pg_write_all_data
   NOLOGIN     pg_monitor
   NOLOGIN     pg_read_all_settings
   NOLOGIN     pg_read_all_stats
   NOLOGIN     pg_stat_scan_tables
   NOLOGIN     pg_read_server_files
   NOLOGIN     pg_write_server_files
   NOLOGIN     pg_execute_server_program
   NOLOGIN     pg_signal_backend
   NOLOGIN     azure_pg_admin
   NOLOGIN     replication
   MD5         nbdbadmin

   To install rdbms-connect extension, ensure that you select the Y option. If installing the extension fails in the az-cli container, then some dependencies must be missing. Install the missing dependencies with apk add gcc musl-dev and try again.

   The nbdbadmin auth type must be SCRAM-256. Resetting the password as follows will re-encrypt the password correctly.

9. Set the new password as follows:

   Before setting the new password ensure that you know your database server name or obtain it by using the following command:

   NEW_DBADMINPASSWORD="<new_password>"

   Use the following command to set the new password:

   az postgres flexible-server execute -p $OLD_DBADMINPASSWORD -u $DBADMINUSER -n $DBSERVER -d postgres -q "ALTER USER\"nbdbadmin\" WITH PASSWORD '$NEW_DBADMINPASSWORD';"

   Or

   If you are only trying to re-encrypt the current password without changing it, use the following command:

   az postgres flexible-server execute -p $OLD_DBADMINPASSWORD -u $DBADMINUSER -n $DBSERVER -d postgres -q "ALTER USER\"nbdbadmin\" WITH PASSWORD '$OLD_DBADMINPASSWORD';"

   Note:

   You can reset the flexible server password by using the following command. This command does not require az extension and potentially could be run outside of the az-cli container.

   az postgres flexible-server update -g $RESOURCE_GROUP -n $DBSERVER --admin-password <password>

10. Use the following command to verify if the password is using the correct encryption method (SCRAM-SHA-256):

    az postgres flexible-server execute -p "$OLD_DBADMINPASSWORD" -u $DBADMINUSER -n $DBSERVER -d postgres -q "SELECT * from azure_roles_authtype();" -o table

    Successfully connected to twilk-db.
    Ran Database Query: 'SELECT * from azure_roles_authtype();'
    Retrieving first 30 rows of query output, if applicable.
    Closed the connection to twilk-db
    Authtype    Rolename
    ----------  -------------------------
    NOLOGIN     azuresu
    NOLOGIN     pg_database_owner
    NOLOGIN     pg_read_all_data
    NOLOGIN     pg_write_all_data
    NOLOGIN     pg_monitor
    NOLOGIN     pg_read_all_settings
    NOLOGIN     pg_read_all_stats
    NOLOGIN     pg_stat_scan_tables
    NOLOGIN     pg_read_server_files
    NOLOGIN     pg_write_server_files
    NOLOGIN     pg_execute_server_program
    NOLOGIN     pg_signal_backend
    NOLOGIN     azure_pg_admin
    NOLOGIN     replication
    SCRAM-256   nbdbadmin

11. Store the updated password in key vault:

    Note:

    This step can be skipped if the password is not changed.

    az keyvault secret set --vault-name $KEY_VAULT_NAME --name dbadminpassword --value "$NEW_DBADMINPASSWORD"

12. (Optional) Delete the key vault access policy created in step 4 above:

    $ az keyvault delete-policy -n $KEYVAULT --upn $USER_ID

13. Exit from the azure CLI pod:

    $ exit

14. Delete the az CLI pod:

    $ kubectl delete pod az-cli

15. (Applicable only for an existing cloudscale deployment) Restart the primary pod:

    $ kubectl rollout restart "statefulset/${PRIMARY}" --namespace "${NAMESPACE}"

    In the above command,

    • NAMESPACE is the namespace containing your NetBackup deployment

    • PRIMARY is the name of primary pod's stateful set

    Use the following command to obtain NAMESPACE and PRIMARY:

    $ kubectl get --namespace "${NAMESPACE}" primaryserver -o jsonpath='{.items[0].status.attributes.resourceName}'

AWS-specific

1. Use lambda function to change the password.

   LAMBDA_ARN is the ARN of the password changing lambda function. This can be obtained from the lambda function page on AWS console.

   NEW_PASSWORD is the new password to be used.

   $ aws lambda invoke --function-name $LAMBDA_ARN \ --cli-binary-format raw-in-base64-out --payload '{"password":"$NEW_PASSWORD"}' \ response_file

2. Wait for database to be available.

   Obtain the POSTGRESQL_ID (database identifier)of your RDS Postgres database from the RDS database page of the AWS console, using the following command:

   $ aws rds wait db-instance-available --db-instance-identifier $POSTGRESQL_ID

3. Restart the primary pod:

   $ kubectl rollout restart "statefulset/${PRIMARY}" --namespace "${NAMESPACE}"

   In the above command,

   • NAMESPACE is the namespace containing your NetBackup deployment

   • PRIMARY is the name of primary pod's stateful set

   Use the following command to obtain NAMESPACE and PRIMARY:

   $ kubectl get --namespace "${NAMESPACE}" primaryserver -o jsonpath='{.items[0].status.attributes.resourceName}'

Containerized PostgreSQL

1. Exec into primary pod and change database password using the following command:

   $ kubectl exec -it <primary-pod-name> -n netbackup -- bash

   # /usr/openv/db/bin/nbdb_admin -dba "<new-password>"

   # exit

2. Update the database connection secret with new password:

   • Set the new password:

     $ kubectl patch secret dbsecret -n netbackup -p '{"stringData": {"dbadminpassword": "<new-password>" }}'

   • Verify the new password:

     $ kubectl get secret dbsecret -n netbackup -o jsonpath='{.data.dbadminpassword}' | base64 --decode

3. Restart the Postgres and primary pods:

   • Identify Postgres and primary statefulsets:

     $ kubectl get statefulset -n netbackup

   • Restart Postgres pod:

     $ kubectl rollout restart "statefulset/nb-postgresql" -n netbackup

   • Wait for the Postgres pod to restart:

     $ kubectl rollout status --watch --timeout=600s "statefulset/nb-postgresql" -n netbackup

   • Restart primary pod:

     $ kubectl rollout restart "statefulset/<primary-statefulset>" -n netbackup

   • Wait for primary pod to restart:

     $ kubectl rollout status --watch --timeout=600s "statefulset/<primary-statefulset>" -n netbackup