Java tracing walkthrough

This section will explain how we implemented tracing using OpenTelemetry for the PetSearch Java service. Multiple getting started examples can be found on the official AWS Distro for OpenTelemetry documentation for Go, Java, Javascript and Python, as well as other platform configuration examples.

Learn more about the OpenTelemetry specification which guides languages implementations here.

X-Ray Setup

All traces generated by the application are collected with the OpenTelemetry collector setup on ECS as sidecar.

ADOT setup

Tracing with AWS auto instrumentation agent

To collect tracing information from a Java application using the auto instrumentation agent, we need to download the Java agent and use the -javaagent flag to the JVM. We can see this setup in the project’s Dockerfile:

ADD https://github.com/aws-observability/aws-otel-java-instrumentation/releases/download/v0.18.0-aws.1/aws-opentelemetry-agent.jar /app/aws-opentelemetry-agent.jar
ENV JAVA_TOOL_OPTIONS "-javaagent:/app/aws-opentelemetry-agent.jar"

By default, OpenTelemetry Java agent uses the OTLP exporter and exports the traces in OTLP format. There are several settings that can be customized for our specific needs using either system properties or environment variables.

In the snippet below, the value of the OTEL_RESOURCE_ATTRIBUTES environment variable we are configuring the service name as PetSearch. This effect of this can be seen on the Service Map.

The value of the OTEL_IMR_EXPORT_INTERVAL environment variable indicates the frequency at which the OTLP metrics will be exported to the OTEL Collector.

The value of the OTEL_EXPORTER_OTLP_ENDPOINT environment variable indicates the endpoint at which the OTEL Collector is listening on. The default configuration for the AWS OTel Collector can be seen here

ENV OTEL_RESOURCE_ATTRIBUTES "service.name=PetSearch"
ENV OTEL_IMR_EXPORT_INTERVAL "10000"
ENV OTEL_EXPORTER_OTLP_ENDPOINT "http://localhost:55680"

PetSearch service is built using Spring framework, and we use the AWS SDK to to make calls to several AWS services. The AWS SDK is supported by OpenTelemetry for auto instrumentation. So any request handled by Spring MVC and the resulting calls to AWS services are automatically traced and published.

The screenshot below shows the metadata captured using auto-instrumentation.

Trace overview

Custom tracing attributes and Spans

When we use the auto instrumentation agent and one of the supported libraries, all the tracing information is already available. But if you are using a library that doesn’t support auto-instrumentation, or if you want to enrich the information provided by default, you can add custom tracing attributes:

Span.current().setAttribute(e.getKey(), e.getValue());

The code below shows how you can add a span. You can see this in the search method in the SearchController.java file

Span span = metricEmitter.spanBuilder("Scanning DynamoDB Table").startSpan();

try(Scope scope = span.makeCurrent()) {

    List<Pet> result = ddbClient.scan(
        buildScanRequest(petType, petColor, petId))
            .getItems().stream().map(this::mapToPet)
            .collect(Collectors.toList());
    return result;

} catch (Exception e) {
    span.recordException(e);
    logger.error("Error while searching, building the resulting body", e);
    throw e;
} finally {
    span.end();
}

The effect of this span can be seen when viewing a trace as shown below Show span