To configure Azure Container insights

1. Enable Azure Container insights.

   See Azure documentation.

2. Download the template ConfigMap YAML file and save it as container-azm-ms-agentconfig.yaml.
3. Add the YAML file with the following sample configuration:

   prometheus-data-collection-settings: |-
   
   [prometheus_data_collection_settings.cluster]
   interval = "1m"
   
   fieldpass = ["msdpoperator_reconcile_total",
                "msdpoperator_reconcile_failed",
                "msdpoperator_operator_run",
                "msdpoperator_diskFreeLess5GBEngines_total",
                "msdpoperator_diskFreeMiBytesInEngine",
                "msdpoperator_diskFreeLess10GBClusters_total",
                "msdpoperator_totalDiskFreePercentInCluster",
                "msdpoperator_diskFreePercentInEngine",
                "msdpoperator_pvcFreePercentInCluster",
                "msdpoperator_unhealthyEngines_total",
                "msdpoperator_createdPods_total"]
   
   monitor_kubernetes_pods = true
   
   # Add the namespace of MSDP operator in the follow list. 
   It's "msdp-operator-system" by default.
   
   monitor_kubernetes_pods_namespaces = 
   ["msdp-operator-system"]

   Table: Supported Prometheus metrics list in MSDP Scaleout lists the Prometheus metrics that MSDP Scaleout supports.

4. Apply the ConfigMap.

   kubectl apply -f container-azm-ms-agentconfig.yaml

   The configuration change takes a few minutes and all omsagent pods in the cluster restart.

   The default namespace of prometheus metrics is prometheus.

5. Add alert rules for the integrated metrics.

   Add related log query, add new alert rule for the selected query, and alert group/action for it.

   For example,

   If the free space size of the MSDP Scaleout engines is lower than 1 GB in past 5 minutes, alert the users.

   Log query:

   InsightsMetrics
   
   | where Name == "msdpoperator_diskFreeMiBytesInEngine"
   | where Namespace == "prometheus"
   | where TimeGenerated > ago(5m)
   | where Val <= 1000000
   | where Val > 0

   If multiple MSDP Scaleouts are deployed in the same AKS cluster, use the filter to search the results. For example, search the MSDP engines with the free space size lower than 1GB in the namespace sample-cr-namespace

   Log query:

   InsightsMetrics
   | where Name == "msdpoperator_diskFreeMiBytesInEngine"
   | where Namespace == "prometheus"
   | where TimeGenerated > ago(5m)
   | where Val <= 1000000
   | where Val > 0
   | extend Tags = parse_json(Tags)
   | where Tags.msdpscalout_ns == "sample-cr-namespace"