CloudWatch Container Insights monitoring for Prometheus automates the discovery of Prometheus metrics from containerized systems and workloads. Prometheus is an open-source systems monitoring and alerting toolkit.
The front end app is already exposing metrics in Prometheus exposition format. Simply add
/metrics to the application URL to see the metrics on the browser.
Before you proceed, we recommend that you setup CloudWatch Container Insights first, in case you haven’t done that already. This will help understand this module better and give you a holistic view of Container Insights as a solution.
On your Cloud9 terminal from
/cdk/pet_stack/ directory, execute the following command. This will deploy the CloudWatch Prometheus agent into the EKS cluster.
kubectl apply -f ./resources/prometheus-eks.yaml
Your output should look similar to the one below:
imaya:~/environment/workshopfiles/PetAdoptions/cdk/pet_stack (master) $ kubectl apply -f ./resources/prometheus-eks.yaml namespace/amazon-cloudwatch unchanged configmap/prometheus-cwagentconfig created configmap/prometheus-config created serviceaccount/cwagent-prometheus created clusterrole.rbac.authorization.k8s.io/cwagent-prometheus-role created clusterrolebinding.rbac.authorization.k8s.io/cwagent-prometheus-role-binding created deployment.apps/cwagent-prometheus created
Execute the following command to make sure the CloudWatch Prometheus agent is running.
kubectl get pods -A
Your output should look similar to the one below. Notice the 3rd line indicating the CloudWatch Prometheus agent running. In the default configuration, there will be just one agent running per cluster.
imaya:~/environment/workshopfiles/PetAdoptions/cdk/pet_stack (master) $ kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE amazon-cloudwatch cloudwatch-agent-ttlmd 1/1 Running 0 5m15s amazon-cloudwatch cloudwatch-agent-tvqcm 1/1 Running 0 5m15s amazon-cloudwatch cwagent-prometheus-7cf89cbb57-bfn9k 1/1 Running 0 4m54s amazon-cloudwatch fluentd-cloudwatch-f5k9g 1/1 Running 0 5m15s amazon-cloudwatch fluentd-cloudwatch-p2c7z 1/1 Running 0 5m15s default petsite-deployment-56c745d764-br8jx 1/1 Running 0 32h default petsite-deployment-56c745d764-x4mft 1/1 Running 0 32h default xray-daemon-m8mr5 1/1 Running 0 32h default xray-daemon-qqxsc 1/1 Running 0 32h kube-system aws-node-9mnsx 1/1 Running 0 32h kube-system aws-node-vrhtc 1/1 Running 0 32h kube-system coredns-5c97f79574-bwhvq 1/1 Running 0 32h kube-system coredns-5c97f79574-xzsw8 1/1 Running 0 32h kube-system kube-proxy-5d5sx 1/1 Running 0 32h kube-system kube-proxy-gp98d 1/1 Running 0 32h
prometheus-eks.yaml file under
/cdk/pet_stack/resources/ directory and search for the term
PetSite to find the configuration used to collect metrics from the application endpoint. The following screenshot shows a portion of the configuration where we specify metric selectors and appropriate dimensions under which the metrics should be aggregated.
Go to CloudWatch Metrics and you should see a new Namespace called
You will see that there are 3 dimensions under which the metrics are being aggregated. Select each one and see the metrics under them.
Just like regular Container Insights metrics, Prometheus metrics is also collected through Embedded Metric Format. Go to CloudWatch Logs which will take you directly to the log group under which the Prometheus metrics are collected in Embedded Metric Format.
PetSite logstream, you should see the Prometheus metrics that were collected from the environment.
Read more about this topic in this blog post - https://aws.amazon.com/blogs/containers/using-prometheus-metrics-in-amazon-cloudwatch/