Prometheus Metrics
CloudWatch Container Insights monitoring for Prometheus automates the discovery of Prometheus metrics from containerized systems and workloads. Prometheus is an open-source systems monitoring and alerting toolkit.
Architecture
The front end app is already exposing metrics in Prometheus exposition format. Simply add /metrics to the application URL to see the metrics on the browser.
Deploy CloudWatch Prometheus agent
Before you proceed, we recommend that you setup CloudWatch Container Insights first, in case you haven’t done that already. This will help understand this module better and give you a holistic view of Container Insights as a solution.
On your Cloud9 terminal from /cdk/pet_stack/ directory, execute the following command. This will deploy the CloudWatch Prometheus agent into the EKS cluster.
kubectl apply -f ./resources/prometheus-eks.yaml
Your output should look similar to the one below:
imaya:~/environment/workshopfiles/PetAdoptions/cdk/pet_stack (master) $ kubectl apply -f ./resources/prometheus-eks.yaml
namespace/amazon-cloudwatch unchanged
configmap/prometheus-cwagentconfig created
configmap/prometheus-config created
serviceaccount/cwagent-prometheus created
clusterrole.rbac.authorization.k8s.io/cwagent-prometheus-role created
clusterrolebinding.rbac.authorization.k8s.io/cwagent-prometheus-role-binding created
deployment.apps/cwagent-prometheus created
Execute the following command to make sure the CloudWatch Prometheus agent is running.
kubectl get pods -A
Your output should look similar to the one below. Notice the 3rd line indicating the CloudWatch Prometheus agent running. In the default configuration, there will be just one agent running per cluster.
imaya:~/environment/workshopfiles/PetAdoptions/cdk/pet_stack (master) $ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
amazon-cloudwatch cloudwatch-agent-ttlmd 1/1 Running 0 5m15s
amazon-cloudwatch cloudwatch-agent-tvqcm 1/1 Running 0 5m15s
amazon-cloudwatch cwagent-prometheus-7cf89cbb57-bfn9k 1/1 Running 0 4m54s
amazon-cloudwatch fluentd-cloudwatch-f5k9g 1/1 Running 0 5m15s
amazon-cloudwatch fluentd-cloudwatch-p2c7z 1/1 Running 0 5m15s
default petsite-deployment-56c745d764-br8jx 1/1 Running 0 32h
default petsite-deployment-56c745d764-x4mft 1/1 Running 0 32h
default xray-daemon-m8mr5 1/1 Running 0 32h
default xray-daemon-qqxsc 1/1 Running 0 32h
kube-system aws-node-9mnsx 1/1 Running 0 32h
kube-system aws-node-vrhtc 1/1 Running 0 32h
kube-system coredns-5c97f79574-bwhvq 1/1 Running 0 32h
kube-system coredns-5c97f79574-xzsw8 1/1 Running 0 32h
kube-system kube-proxy-5d5sx 1/1 Running 0 32h
kube-system kube-proxy-gp98d 1/1 Running 0 32h
CloudWatch Prometheus agent configuration
Open the prometheus-eks.yaml file under /cdk/pet_stack/resources/ directory and search for the term PetSite to find the configuration used to collect metrics from the application endpoint. The following screenshot shows a portion of the configuration where we specify metric selectors and appropriate dimensions under which the metrics should be aggregated.
See metrics collected by the CloudWatch Prometheus agent
Go to CloudWatch Metrics and you should see a new Namespace called ContainerInsights/Prometheus.
You will see that there are 3 dimensions under which the metrics are being aggregated. Select each one and see the metrics under them.
Just like regular Container Insights metrics, Prometheus metrics is also collected through Embedded Metric Format. Go to CloudWatch Logs which will take you directly to the log group under which the Prometheus metrics are collected in Embedded Metric Format.
Under PetSite logstream, you should see the Prometheus metrics that were collected from the environment.
Read more about this topic in this blog post - https://aws.amazon.com/blogs/containers/using-prometheus-metrics-in-amazon-cloudwatch/
Autoscaling Amazon EKS services based on custom Prometheus metrics using CloudWatch Container Insights
Take a look at this blog post from Viji Sarathy that talks about performing autoscaling actions using Prometheus metrics collected using the Cloudwatch Prometheus agent.