Prometheus Metrics

CloudWatch Container Insights monitoring for Prometheus automates the discovery of Prometheus metrics from containerized systems and workloads. Prometheus is an open-source systems monitoring and alerting toolkit.

Architecture

CloudWatch Prometheus agent

The front end application is already exposing metrics in Prometheus exposition format. By adding /metrics to the application URL, you can view the metrics in the browser.

CloudWatch Prometheus Agent

  1. In the AWS Management Console on the Services menu, click Cloud9.
  2. Click Open IDE on the observabilityworkshop Cloud9 instance.
  3. Execute the following command in the terminal:

This command ensures that the CloudWatch Prometheus agent is running.

kubectl get pods -n amazon-cloudwatch

Your output should look similar to the one below. (Notice the 3rd line indicating the CloudWatch Prometheus agent running. In the default configuration, there will be just one agent running per cluster.)

NAME                                  READY   STATUS    RESTARTS   AGE
cloudwatch-agent-fnh25                1/1     Running   0          172m
cloudwatch-agent-tfgk7                1/1     Running   0          172m
cwagent-prometheus-5bbf77558b-sqkfm   1/1     Running   0          172m
fluent-bit-7p67r                      1/1     Running   0          172m
fluent-bit-cqphf                      1/1     Running   0          172m

CloudWatch Prometheus agent configuration

  1. Open the file called prometheus-eks.yaml located at one-observability-demo/PetAdoptions/cdk/pet_stack/resources/prometheus-eks.yaml (or alternatively, click here).

  2. Search the file for the term `PetSite.

Here you will find the configurations used to collect metrics from the application endpoint. The following screenshot shows a portion of the configuration where we specify metric selectors and appropriate dimensions under which the metrics should be aggregated.

CloudWatch Prometheus agent

See metrics collected by the CloudWatch Prometheus agent

  1. Navigate to CloudWatch in the AWS Console.
  2. In the left navigation menu, select Metrics.

You should see a new namespace called ContainerInsights/Prometheus under Custom Namespaces.

CloudWatch Prometheus agent

  1. Click on the ContainerInsights/Prometheus that was just created.
  2. Click ClusterName, Namespace, app.
  3. Check the checkboxes of a few of the metrics in the list and watch them appear on the graph.

CloudWatch Prometheus agent

Just like regular Container Insights metrics, Prometheus metrics are also collected through Embedded Metric Format.

  1. In the left navigation menu under logs, click log groups.
  2. In the filter log groups search box, type prometheus.
  3. Select the /aws/containerinsights/PetSite/prometheus.

Alternatively, you can click here to go directly to the log group under which the Prometheus metrics are collected.

  1. Click on the PetSite log stream.

You should see the Prometheus metrics that were collected from the environment.

CloudWatch Prometheus agent

  1. Click on an arrow from the first column in the table to expand a log.

CloudWatch Prometheus agent

Read more about this topic in this blog post - https://aws.amazon.com/blogs/containers/using-prometheus-metrics-in-amazon-cloudwatch/

Autoscaling Amazon EKS Services based on custom Prometheus metrics using CloudWatch Container Insights

Take a look at this blog post from Viji Sarathy that talks about performing autoscaling actions using Prometheus metrics collected using the Cloudwatch Prometheus agent.