Since its launch in May 2019, the OpenTelemetry (OTel) Project has become one of the most popular open source projects since Kubernetes®. For newcomers to the observability domain, OpenTelemetry™ provides a standard way to collect telemetry data (metric, logs and traces) from software applications and send it to one or more backends to analyze application performance. The backends can be open source (Jaeger or Zipkin, for example), commercial (such as AppDynamics Cloud or AppDynamics APM) or both.
To enable faster adoption and showcase instrumentation best practices, the OTel community recently built a demo application, OpenTelemetry Community Demo. In this blog, I’ll show how to configure the OpenTelemetry demo to send telemetry data to AppDynamics Cloud for further analysis.
Backstory: AppDynamics Cloud and OpenTelemetry demo
AppDynamics Cloud, our new software-as-a-service (SaaS) solution, provides full-stack observability of cloud native applications and natively supports OpenTelemetry-based ingestion.
In addition to the microservices shown here, the demo app also comes with supporting components such as OpenTelemetry Collector, Grafana, Prometheus and Jaeger to export and visualize traces, metrics and so on. OpenTelemetry Collector can be configured to send telemetry data to multiple backends simultaneously. The diagram below shows the OTel demo with supporting components, as well as a dotted line to AppDynamics Cloud, which we will configure in the next section.
Sending telemetry data to AppDynamics Cloud
- Using the steps described in the Open Telemetry demo Docker deployment documentation, deploy the demo app on your local machine.
- Contact your AppDynamics account representative to set up an AppDynamics tenant for your company. The tenant will have a URL format similar to https://<your-company-name>.observe.appdynamics.com and will be the central location where you’ll see all telemetry data from your applications.
- For this Docker setup, we’ll use the credentials generated for Kubernetes and APM monitoring. To generate credentials, see the steps highlighted in the AppDynamics UI below.
- We will use the following key-values:
- As described below, update the file src/otelcollector/otelcol-config-extras.yml in your cloned repo. The contents of this file get merged with src/otelcollector/otelcol-config.yml at runtime to create the consolidated OpenTelemetry Collector configuration.
# AppD cloud requires enabling oath2client extension for sending data to it
client_id: <from step 4a above>
client_secret: <from step 4b above >
token_url: <from step 4d above>
traces_endpoint: <from step 4c above>/v1beta/trace
metrics_endpoint: <from step 4c above>/v1beta/metrics
– key: service.namespace
processors: [batch, resource]
exporters: [logging, otlp, otlphttp/appd]
processors: [batch, resource]
exporters: [logging, prometheus, otlphttp/appd]
- Stop the docker containers (‘docker compose down’) and then start them (‘ docker compose up -d’). Wait a few minutes and confirm that you can access the OpenTelemetry demo app UI at http://localhost:8080/. Next, log in to your AppDynamics Cloud Tenant URL and click the “Service Topology” shortcut button. You’ll then see a service flow map that shows various microservices and the interactions between them.
- Click the List view to display key APM metrics such as Avg Response Time, Calls/min and Errors/min.
- An observability platform should be able to detect an increase in error rates of the microservices it is monitoring. Fortunately, the OpenTelemetry demo has an error injection capability to test this functionality. Go to the feature flag UI at http://localhost:8080/feature/ and enable the productCatalogFailure feature flag. This will cause the product catalog service to return an error for a specific product ID and respond correctly to all other product IDs. Note the increase in error rate captured by AppDynamics Cloud. In AppDynamics, you can set up health rules and alerts to respond quickly to such situations.
OTel and AppDynamics Cloud
The OpenTelemetry Community Demo application is a valuable and safe tool for learning about OpenTelemetry and instrumentation best practices. In this blog, we showed how to configure the demo app to send telemetry data to AppDynamics Cloud. We also explored some key AppDynamics Cloud features such as the flow map, APM metrics, and an observed increase in error rates via a fault-injection scenario.