To configure Amazon CloudWatch

1. Install the CloudWatch agent with Prometheus metrics collection on EKS.

   See AWS documentation.

2. Install the CloudWatch agent on EKS clusters. Select the EC2 launch type, and download the template YAML file Prometheus-eks.yaml.
3. Add the YAML file with the following sample configuration.

   # create configmap for prometheus cwagent config
   apiVersion: v1
   data:
     # cwagent json config
     cwagentconfig.json: |
       {
         "logs": {
           "metrics_collected": {
             "prometheus": {
               "prometheus_config_path": "/etc/prometheusconfig/
                prometheus.yaml",
               "emf_processor": {
                 "metric_declaration": [
                   {
                     "source_labels": ["job"],
                     "label_matcher": "^msdpoperator-metrics",
                     "dimensions":[
                       ["ClusterName","NameSpace"]
                     ],
                     "metric_selectors": [
                       "msdpoperator_reconcile_failed",
                       "msdpoperator_operator_run",
                       "msdpoperator_diskFreeLess5GBEngines_total",
                       "msdpoperator_diskFreeMiBytesInEngine",
                       "msdpoperator_diskFreeLess10GBClusters_total",
                       "msdpoperator_totalDiskFreePercentInCluster",
                       "msdpoperator_diskFreePercentInEngine",
                       "msdpoperator_pvcFreePercentInCluster",
                       "msdpoperator_unhealthyEngines_total",
                       "msdpoperator_createdPods_total"
                     ]
                   }
                 ]
               }
             }
           },
           "force_flush_interval": 5
         }
       }
   kind: ConfigMap
   metadata:
     name: prometheus-cwagentconfig
     namespace: amazon-cloudwatch
    
   ---
   # create configmap for prometheus scrape config
   apiVersion: v1
   data:
     # prometheus config
     prometheus.yaml: |
       global:
         scrape_interval: 1m
         scrape_timeout: 10s
       scrape_configs:
       - job_name: 'msdpoperator-metrics'
    
         scheme: https
         tls_config:
           ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
           insecure_skip_verify: true
         bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount
         /token
    
         kubernetes_sd_configs:
         - role: pod
    
         relabel_configs:
         - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io
           _scrape]
           action: keep
           regex: true
         - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io
           _path]
           action: replace
           target_label: __metrics_path__
           regex: (.+)
         - source_labels: [__address__, __meta_kubernetes_pod_annotation_
           prometheus_io_port]
           action: replace
           regex: ([^:]+)(?::\d+)?;(\d+)
           replacement: $1:$2
           target_label: __address__
         - source_labels: [__meta_kubernetes_namespace]
           action: replace
           target_label: NameSpace
         - source_labels: [__meta_kubernetes_pod_name]
           action: replace
           target_label: PodName
    
   kind: ConfigMap
   metadata:
     name: prometheus-config
     namespace: amazon-cloudwatch
   

   Table: Supported Prometheus metrics list in MSDP Scaleout lists the Prometheus metrics that MSDP Scaleout supports.

4. Apply the YAML file.

   Kubectl apply -f Prometheus-eks.yaml

   The default log groups name is /aws/containerinsights/{cluster_name}/Prometheus.

5. Create Amazon CloudWatch alarms.

   See Using Amazon CloudWatch alarms in AWS documentation.

6. In the CloudWatch console, add the related log query. In the navigation pane, select Log Insights.

   For example, the free space size of the MSDP scaleout cluster engines is lower than 1 GB in past 5 minutes. Select the log group from the drop-down list, and select the time duration 5m on the time line.

   Log query:

   fields @timestamp, @message
   | filter msdpoperator_diskFreeMiBytesInEngine <= 100000
   | sort @timestamp desc
   

   If multiple MSDP scaleout clusters are deployed in the same EKS cluster, use the filter to search the results. For example, search the MSDP engines with the free space size lower than 1GB in the namespace sample-cr-namespace.

   Log query:

   fields @timestamp, @message
   | filter msdpscalout_ns == "sample-cr-namespace"
   | filter msdpoperator_diskFreeMiBytesInEngine <= 100000
   | sort @timestamp desc