Ingest metrics from EKS

Amazon Managed Service for Prometheus does not directly scrape operational metrics from containerized workloads in a Kubernetes cluster. It requires users to deploy and manage a standard Prometheus server, Grafana Cloud Agent or an OpenTelemetry agent such as the AWS Distro for OpenTelemetry Collector in their cluster to perform this task.

In this section, we will go through how you can deploy a Prometheus server to scrape and ingest metrics into AMP. Take a look at this blog post for instructions to use Grafana Cloud Agent as an alternate option.

Deploy Prometheus server

Install Helm

Install helm using instructions from the public site, or execute the steps below.

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

Add new Helm chart repositories

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add kube-state-metrics https://kubernetes.github.io/kube-state-metrics
helm repo update

Setup IAM Role and Kubernetes Service Account

Execute the commands shown below. It creates an IAM role named EKS-AMP-ServiceAccount-Role and attaches it to the Kubernetes service account named amp-iamproxy-service-account under the prometheus and grafana namespaces.

kubectl create ns prometheus
chmod +x ./resources/amp-setup-irsa-eks.sh
./resources/amp-setup-irsa-eks.sh

Set up the new server and start ingesting metrics

Before you execute the helm install command, follow the steps below.

  • Replace my_prometheus_iam_proxy_role_arn with the ARN of the amp-iamproxy-ingest-role that you created in the previous step

Execute the following command to get the Role ARN

aws iam get-role --role-name amp-iamproxy-ingest-role | jq .Role.Arn
  • Replace <my_workspace_id> with the Workspace ID of your AMP workspace.

This command will list all workspaces in your account. Pick the one that you want to use.

aws amp list-workspaces
  • Replace <my_workspace_region> with the AWS Region of your AMP workspace. [Eg. us-east-1]
IAM_PROXY_PROMETHEUS_ROLE_ARN=<my_prometheus_iam_proxy_role_arn>
WORKSPACE_ID=<my_workspace_id>
AWS_REGION=<my_workspace_region>

Once the environment variables are set, execute the following command

helm install amp-prometheus-chart prometheus-community/prometheus -n prometheus -f ./resources/amp_ingest_override_values.yaml \
--set serviceAccounts.server.annotations."eks\.amazonaws\.com/role-arn"="${IAM_PROXY_PROMETHEUS_ROLE_ARN}" \
--set server.remoteWrite[0].url="https://aps-workspaces.${AWS_REGION}.amazonaws.com/workspaces/${WORKSPACE_ID}/api/v1/remote_write" \
--set server.remoteWrite[0].sigv4.region=${AWS_REGION}

Your setup now looks similar to the architecture diagram below

The VPC Endpoint shown in the diagram is optional and is not used in the instructions here

AMP setup

Setup ADOT Collector

The AWS Distro for the OpenTelemetry Collector is another option for you to use to ingest metrics into AMP. The ADOT-AMP pipeline enables us to use the ADOT Collector to scrape a Prometheus-instrumented application, and send the scraped metrics to Amazon Managed Service for Prometheus (AMP).

ADOT-AMP pipeline

Open the ./resources/amp-eks-adot-prometheus-daemonset.yaml file and do the following:

  • Replace the text <AMP_REMOTE_WRITE_URL> with the Remote Write URL from the workspace you just created
  • Replace the text <AWS_ACCOUNT_ID> with the AWS ACCOUNT ID
  • Replace the text <AWS_REGION> with the AWS Region of the AMP workspace

Deploy the ADOT collector using the following command.

kubectl apply -f ./resources/amp-eks-adot-prometheus-daemonset.yaml

Validate the setup

Execute following commands to see the list of Pods in the cluster

kubectl get pods -n prometheus

Your result should look similar to the one below. As you can see there are is the Prometheus server deployed as a Pod in the EKS cluster

NAME                                                       READY   STATUS    RESTARTS   AGE
adot-collector-hxglz                                     1/1     Running   0          12s
adot-collector-w4f4b                                     1/1     Running   0          12s
amp-prometheus-chart-kube-state-metrics-579888d7-nf546   1/1     Running   0          11m
amp-prometheus-chart-node-exporter-mww2k                 1/1     Running   0          11m
amp-prometheus-chart-node-exporter-zmpk7                 1/1     Running   0          11m
amp-prometheus-chart-server-0                            2/2     Running   0          11m

The ADOT collector and the Prometheus server sign the requests when ingesting into the AMP workspace.

Execute the following command to see the Prometheus interface. This allows you to connect to the Prometheus server container from localhost.

kubectl port-forward -n prometheus pods/amp-prometheus-chart-server-0 8080:9090

Now navigate to http://127.0.0.1:8080 to see the prometheus interface.

If you are on Cloud9, you can open the preview browser by clicking on Preview Running Application as shown below.

Prom dashboard

Paste the following PromQL in the query text box, click Execute and switch to the Graph tab to see the results as shown below

rate(node_network_receive_bytes_total[5m])

Prom dashboard

Go to the Configuration page as shown below and search for the keyword remote where you will find that the remote_write destination has been set to the AMP workspace as shown below.

Configuration Page

Prom dashboard