Monitoring with Azure Container insights
You can use Azure Container insights to collect Prometheus metrics to monitor pods in MSDP Scaleout.
To configure Azure Container insights
- Enable Azure Container insights.
See Azure documentation.
- Download the template ConfigMap YAML file and save it as
container-azm-ms-agentconfig.yaml. - Add the YAML file with the following sample configuration:
prometheus-data-collection-settings: |- [prometheus_data_collection_settings.cluster] interval = "1m" fieldpass = ["msdpoperator_reconcile_total", "msdpoperator_reconcile_failed", "msdpoperator_operator_run", "msdpoperator_diskFreeLess5GBEngines_total", "msdpoperator_diskFreeMiBytesInEngine", "msdpoperator_diskFreeLess10GBClusters_total", "msdpoperator_totalDiskFreePercentInCluster", "msdpoperator_diskFreePercentInEngine", "msdpoperator_pvcFreePercentInCluster", "msdpoperator_unhealthyEngines_total", "msdpoperator_createdPods_total"] monitor_kubernetes_pods = true # Add the namespace of MSDP operator in the follow list. It's "msdp-operator-system" by default. monitor_kubernetes_pods_namespaces = ["msdp-operator-system"]Table: Supported Prometheus metrics list in MSDP Scaleout lists the Prometheus metrics that MSDP Scaleout supports.
- Apply the ConfigMap.
kubectl apply -f container-azm-ms-agentconfig.yaml
The configuration change takes a few minutes and all omsagent pods in the cluster restart.
The default namespace of prometheus metrics is prometheus.
- Add alert rules for the integrated metrics.
Add related log query, add new alert rule for the selected query, and alert group/action for it.
For example,
If the free space size of the MSDP Scaleout engines is lower than 1 GB in past 5 minutes, alert the users.
Log query:
InsightsMetrics | where Name == "msdpoperator_diskFreeMiBytesInEngine" | where Namespace == "prometheus" | where TimeGenerated > ago(5m) | where Val <= 1000000 | where Val > 0
If multiple MSDP Scaleouts are deployed in the same AKS cluster, use the filter to search the results. For example, search the MSDP engines with the free space size lower than 1GB in the namespace sample-cr-namespace
Log query:
InsightsMetrics | where Name == "msdpoperator_diskFreeMiBytesInEngine" | where Namespace == "prometheus" | where TimeGenerated > ago(5m) | where Val <= 1000000 | where Val > 0 | extend Tags = parse_json(Tags) | where Tags.msdpscalout_ns == "sample-cr-namespace"
MSDP Scaleout supports the following Prometheus metrics:
Table: Supported Prometheus metrics list in MSDP Scaleout
Metrics | Type | Filters | Description |
|---|---|---|---|
msdpoperator_reconcile_total | Counter | N/A | The total of the reconcile loops msdp-operator run. |
msdpoperator_reconcile_failed | Counter | N/A | The total of the reconcile loops msdp-operator failed to run. |
msdpoperator_operator_run | Counter | N/A | The total of the running operator. |
msdpoperator_diskFreeLess5GBEngines_total | Gauge | InsightsMetrics.Tags.msdpscalout_ns | The checked number of the engines which have free spaces lower than 5GB. |
msdpoperator_diskFreeMiBytesInEngine | Gauge | InsightsMetrics.Tags.msdpscalout_ns | The free space of current engine in MiBytes. |
msdpoperator_diskFreeLess10GBClusters_total | Gauge | InsightsMetrics.Tags.msdpscalout_ns | The checked number of the msdpscaleout apps which have free spaces lower than 10GB. |
msdpoperator_totalDiskFreePercentInCluster | Gauge | InsightsMetrics.Tags.msdpscalout_ns | The percent of the msdpscaleout apps that have free spaces. For example, 0.95 means 95% |
msdpoperator_diskFreePercentInEngine | Gauge | InsightsMetrics.Tags.msdpscalout_ns | The percent of the current engines, which have free spaces. |
msdpoperator_pvcFreePercentInCluster | Gauge | InsightsMetrics.Tags.msdpscalout_ns, InsightsMetrics.Tags.component | The percent of the used PVC, which have free spaces. |
msdpoperator_unhealthyEngines_total | Gauge | InsightsMetrics.Tags.msdpscalout_ns | The total of unhealthy engines. |
msdpoperator_createdPods_total | Gauge | InsightsMetrics.Tags.msdpscalout_ns, InsightsMetrics.Tags.component | The total of created msdpscaleout pods. |