NodeJS Library

Generate CloudWatch Metrics embedded within structured log events. The embedded metrics will be extracted so you can visualize and alarm on them for real-time incident detection. This allows you to monitor aggregated values while preserving the detailed event context that generated them.

Using the Embedded Metric Format, you will be able to visualize and alarm on custom metrics, but also retain the original, detailed and high-cardinality context which is queryable using CloudWatch Logs Insights. For example, the library automatically injects environment metadata such as Lambda Function version, EC2 instance and image IDs into the structured log event data.

  1. Open the Cloud9 environment you used when configuring the EKS micro-service application.

  2. Open the “AWS Resources” menu on the right hand side and click the Lambda icon to create a new serverless application.

  3. Call the function name emfTestFunction and the application name emfTestFunction and click Next.

  4. For the blueprint select empty-nodejs and click next.

  5. When it asks for Function trigger select none and click next.

  6. When it asks for memory and role leave the defaults and click next and then Finish.

    EMF1

  7. Now click Window and “New Terminal”.

  8. Type and execute the following commands

    cd emfTestFunction/emfTestFunction
    npm init
    
  9. Accept all the defaults (keep hitting return until it asks if this is OK and type yes).

  10. Now execute the following command to install the client library

    npm install aws-embedded-metrics
    
  11. Now open the index.js file from the Environment tree on the left and copy & paste the following text replacing the existing text.

    const { metricScope } = require("aws-embedded-metrics");
    
    const aggregator = metricScope(metrics => async event => {
      console.log("received message");
      metrics.putDimensions({ Service: "Aggregator" });
      metrics.putMetric("ProcessingLatency", 100, "Milliseconds");
      metrics.setProperty("AccountId", "123456789012");
      metrics.setProperty("RequestId", "422b1569-16f6-4a03-b8f0-fe3fd9b100f8");
      metrics.setProperty("DeviceId", "61270781-c6ac-46f1-baf7-22c808af8162");
      metrics.setProperty("Payload", {
        sampleTime: 123456789,
        temperature: 273.0,
        pressure: 101.3
      });
      console.log("completed aggregation successfully.");
    });
    
    exports.handler = aggregator;
    
  12. Right click on the emfTestFunction on the Lambda pane on the right and click Deploy.

  13. Navigate to Lambda in the AWS Management Console and find your function which should have a name that starts with cloud9-emfTestFunction-emfTestFunction-.

  14. Once you have located your Lambda function click the Test button at the top and specify an Event name and click Create.

  15. Click the Test button several times (5-10 times is sufficent), each time you hit Test you should see a box similar to this one:

  16. Now navigate to CloudWatch in the AWS Management Console and click Metrics where you should see a aws-embedded-metrics box under Custom Namespaces. Upon clicking that you will see another box that should say LogGroup, Service, ServiceName, ServiceType which you should click as well.

  17. What you will see is a metric with a Dimension of Aggregator under Service column which maps to this code:

      metrics.putDimensions({ Service: "Aggregator" });
    
  18. Youll also see the Metric Name is ProcessingLatency and when you put the metric on the graph youll see the value of 100ms which maps to this code:

      metrics.putMetric("ProcessingLatency", 100, "Milliseconds");
    

    Example of the metric:

  19. So what about these lines of code? Where are they?

     metrics.setProperty("AccountId", "123456789012");
      metrics.setProperty("RequestId", "422b1569-16f6-4a03-b8f0-fe3fd9b100f8");
      metrics.setProperty("DeviceId", "61270781-c6ac-46f1-baf7-22c808af8162");
      metrics.setProperty("Payload", {
        sampleTime: 123456789,
        temperature: 273.0,
        pressure: 101.3
      });
    
  20. Properties are stored in the logs that are stored in CloudWatch Log Groups along with the other information such as the metric name/value.

  21. Navigate to CloudWatch Log Insights(https://console.aws.amazon.com/cloudwatch/home?#logsV2:log-groups/log-group/) and select the Log Group that starts with /aws/lambda/cloud9-TestFn-TestFn- and click Run Query.

  22. You will see several different logs from the Lambda function so let`s filter to the logs with the metric data with this query.

    fields @timestamp, ProcessingLatency, AccountId, RequestId, Payload.pressure, Payload.sampleTime, Payload.temperature
    | sort @timestamp desc
    | limit 20
    | filter Service='Aggregator'
    
  23. You should now see results that look simliar to this screenshot:

  24. To recap, CloudWatch embedded metric format is a JSON specification used to instruct CloudWatch Logs to automatically extract metric values embedded in structured log events. You can use CloudWatch to graph and create alarms on the extracted metric values.