Golang tracing walkthrough

Multiple getting started examples can be found on the official AWS Distro for OpenTelemetry documentation for Go, Java, Javascript and Python, as well as other platform configuration examples. This section will explain how we implemented tracing using OpenTelemetry for one of our Go microservice in charge of listing pet adoptions.

Learn more about the OpenTelemetry specification which guides languages implementations here.

X-Ray setup

All traces generated by the application are collected with the OpenTelemetry collector setup on ECS as sidecar.

ADOT setup

HTTP server tracing

To automatically trace all API endpoints, we used the powerful gorilla/mux HTTP multiplexer. It provides an OpenTelemetry instrumentation library to wrap around the HTTP handlers. All calls made to those endpoints will be automatically traced and sent to AWS X-Ray according to the sampling rules in place.

r := mux.NewRouter()
r.Use(otelmux.Middleware("petlistadoptions"))

r.Methods("GET").Path("/api/adoptionlist/").Handler(handlerFunc)

HTTP server

To use otelmux, you will need those go packages as dependencies: go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux go get github.com/gorilla/mux

HTTP client tracing

Applications often depend on other services via HTTP APIs. We want also to enable visibility on the client calls. To list all the adoptions, we need information that is exposed via the PetSearch service. Using go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp, all calls to the PetSearch service are captured as remote resources and merged together.

client := http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}

req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
resp, err := client.Do(req)
if err != nil {
  level.Error(logger).Log("err", err)
  return
}

HTTP client

Custom tracing attributes

OpenTelemetry provides various tracing implementations for popular go packages such as otelhttp shown above. However, we could have a specific use case, or some packages may not available yet. To accomodate these use cases, you can do custom tracing using Spans and Go context. The following example shows custom tracing around database queries, with custom business attributes attached to the traces.

tracer := otel.GetTracerProvider().Tracer("petlistadoptions")
_, span := tracer.Start(ctx, "PGSQL Query", trace.WithSpanKind(trace.SpanKindClient))

sql := `SELECT pet_id, transaction_id, adoption_date FROM transactions ORDER BY id DESC LIMIT 25`

// injecting custom attributes
span.SetAttributes(
  label.String("sql", sql),
  label.String("url", r.safeConnStr),
)

rows, err := r.db.Query(sql)
if err != nil {
  handleErr(err)
}
span.End()

Custom tracing

OpenTelemetry attributes are displayed as Metadata

This concludes this section. You may continue on to the next section.