Prometheus Metrics

CloudWatch Container Insights monitoring for Prometheus automates the discovery of Prometheus metrics from containerized systems and workloads. Prometheus is an open-source systems monitoring and alerting toolkit.

Architecture

CloudWatch Prometheus agent

The front end app is already exposing metrics in Prometheus exposition format. Simply add /metrics to the application URL to see the metrics on the browser.

Deploy CloudWatch Prometheus agent

Before you proceed, we recommend that you setup CloudWatch Container Insights first, in case you haven’t done that already. This will help understand this module better and give you a holistic view of Container Insights as a solution.

On your Cloud9 terminal from /cdk/pet_stack/ directory, execute the following command. This will deploy the CloudWatch Prometheus agent into the EKS cluster.

kubectl apply -f ./resources/prometheus-eks.yaml 

Your output should look similar to the one below:

imaya:~/environment/workshopfiles/PetAdoptions/cdk/pet_stack (master) $ kubectl apply -f ./resources/prometheus-eks.yaml                            
namespace/amazon-cloudwatch unchanged
configmap/prometheus-cwagentconfig created
configmap/prometheus-config created
serviceaccount/cwagent-prometheus created
clusterrole.rbac.authorization.k8s.io/cwagent-prometheus-role created
clusterrolebinding.rbac.authorization.k8s.io/cwagent-prometheus-role-binding created
deployment.apps/cwagent-prometheus created

Execute the following command to make sure the CloudWatch Prometheus agent is running.

kubectl get pods -A

Your output should look similar to the one below. Notice the 3rd line indicating the CloudWatch Prometheus agent running. In the default configuration, there will be just one agent running per cluster.

imaya:~/environment/workshopfiles/PetAdoptions/cdk/pet_stack (master) $ kubectl get pods -A
NAMESPACE           NAME                                  READY   STATUS    RESTARTS   AGE
amazon-cloudwatch   cloudwatch-agent-ttlmd                1/1     Running   0          5m15s
amazon-cloudwatch   cloudwatch-agent-tvqcm                1/1     Running   0          5m15s
amazon-cloudwatch   cwagent-prometheus-7cf89cbb57-bfn9k   1/1     Running   0          4m54s
amazon-cloudwatch   fluentd-cloudwatch-f5k9g              1/1     Running   0          5m15s
amazon-cloudwatch   fluentd-cloudwatch-p2c7z              1/1     Running   0          5m15s
default             petsite-deployment-56c745d764-br8jx   1/1     Running   0          32h
default             petsite-deployment-56c745d764-x4mft   1/1     Running   0          32h
default             xray-daemon-m8mr5                     1/1     Running   0          32h
default             xray-daemon-qqxsc                     1/1     Running   0          32h
kube-system         aws-node-9mnsx                        1/1     Running   0          32h
kube-system         aws-node-vrhtc                        1/1     Running   0          32h
kube-system         coredns-5c97f79574-bwhvq              1/1     Running   0          32h
kube-system         coredns-5c97f79574-xzsw8              1/1     Running   0          32h
kube-system         kube-proxy-5d5sx                      1/1     Running   0          32h
kube-system         kube-proxy-gp98d                      1/1     Running   0          32h

CloudWatch Prometheus agent configuration

Open the prometheus-eks.yaml file under /cdk/pet_stack/resources/ directory and search for the term PetSite to find the configuration used to collect metrics from the application endpoint. The following screenshot shows a portion of the configuration where we specify metric selectors and appropriate dimensions under which the metrics should be aggregated.

CloudWatch Prometheus agent

See metrics collected by the CloudWatch Prometheus agent

Go to CloudWatch Metrics and you should see a new Namespace called ContainerInsights/Prometheus.

CloudWatch Prometheus agent

You will see that there are 3 dimensions under which the metrics are being aggregated. Select each one and see the metrics under them.

CloudWatch Prometheus agent

Just like regular Container Insights metrics, Prometheus metrics is also collected through Embedded Metric Format. Go to CloudWatch Logs which will take you directly to the log group under which the Prometheus metrics are collected in Embedded Metric Format.

CloudWatch Prometheus agent

Under PetSite logstream, you should see the Prometheus metrics that were collected from the environment.

CloudWatch Prometheus agent

Read more about this topic in this blog post - https://aws.amazon.com/blogs/containers/using-prometheus-metrics-in-amazon-cloudwatch/