How to observe OpenTelemetry demo app in AppDynamics Cloud

November 17 2022
 

Here's how to configure the OpenTelemetry Community Demo application to send telemetry data to AppDynamics Cloud.


Since its launch in May 2019, the OpenTelemetry (OTel) Project has become one of the most popular open source projects since Kubernetes®. For newcomers to the observability domain, OpenTelemetry™ provides a standard way to collect telemetry data (metric, logs and traces) from software applications and send it to one or more backends to analyze application performance. The backends can be open source (Jaeger or Zipkin, for example), commercial (such as AppDynamics Cloud or AppDynamics APM) or both.

To enable faster adoption and showcase instrumentation best practices, the OTel community recently built a demo application, OpenTelemetry Community Demo. In this blog, I’ll show how to configure the OpenTelemetry demo to send telemetry data to AppDynamics Cloud for further analysis.

Backstory: AppDynamics Cloud and OpenTelemetry demo

AppDynamics Cloud, our new software-as-a-service (SaaS) solution, provides full-stack observability of cloud native applications and natively supports OpenTelemetry-based ingestion.

OpenTelemetry Community Demo is a simulated version of an eCommerce store selling astronomy equipment. The app consists of 14+ microservices communicating with each other via HTTP or GRPC. The microservices are built using a variety of programming languages (Java, Javascript, C#, etc.) and instrumented using OpenTelemetry (auto, manual or both). The diagram below shows the data flow and programming languages used.

Image credit: OpenTelemetry Demo contributors.

In addition to the microservices shown here, the demo app also comes with supporting components such as OpenTelemetry Collector, Grafana, Prometheus and Jaeger to export and visualize traces, metrics and so on. OpenTelemetry Collector can be configured to send telemetry data to multiple backends simultaneously. The diagram below shows the OTel demo with supporting components, as well as a dotted line to AppDynamics Cloud, which we will configure in the next section.

 

Sending telemetry data to AppDynamics Cloud

        1. Using the steps described in the Open Telemetry demo Docker deployment documentation, deploy the demo app on your local machine.
        2. Contact your AppDynamics account representative to set up an AppDynamics tenant for your company. The tenant will have a URL format similar to https://<your-company-name>.observe.appdynamics.com and will be the central location where you’ll see all telemetry data from your applications.
        3. For this Docker setup, we’ll use the credentials generated for Kubernetes and APM monitoring. To generate credentials, see the steps highlighted in the AppDynamics UI below.
        4. We will use the following key-values:
          a. clientId:<value>
          b. clientSecret:<value>
          c. endpoint:<value>
          d. tokenUrl:<value>
        5. As described below, update the file src/otelcollector/otelcol-config-extras.yml in your cloned repo. The contents of this file get merged with src/otelcollector/otelcol-config.yml at runtime to create the consolidated OpenTelemetry Collector configuration.

          extensions:

          # AppD cloud requires enabling oath2client extension for sending data to it
            oauth2client/appd:
              client_id: <from step 4a above>
              client_secret: <from step 4b above >
              token_url: <from step 4d above>
           
          exporters:
            otlphttp/appd:  
              traces_endpoint: <from step 4c above>/v1beta/trace
              metrics_endpoint: <from step 4c above>/v1beta/metrics      
              auth:
                authenticator: oauth2client/appd

          processors:
            batch:
            resource:
              attributes:
              – key: service.namespace
                action: upsert  
                value: otel-demo

          service:
            extensions: [oauth2client/appd]
            pipelines:
              traces:
                receivers: [otlp]
                processors: [batch, resource]
                exporters: [logging, otlp, otlphttp/appd]
              metrics:
                receivers: [otlp]
                processors: [batch, resource]
                exporters: [logging, prometheus, otlphttp/appd]
        6. Stop the docker containers (‘docker compose down’)  and then start them  (‘ docker compose up -d’). Wait a few minutes and confirm that you can access the OpenTelemetry demo app UI at http://localhost:8080/. Next, log in to your AppDynamics Cloud Tenant URL and click the “Service Topology” shortcut button. You’ll then see a service flow map that shows various microservices and the interactions between them.
        7. Click the List view to display key APM metrics such as Avg Response Time, Calls/min and Errors/min.
        8. An observability platform should be able to detect an increase in error rates of the microservices it is monitoring. Fortunately, the OpenTelemetry demo has an error injection capability to test this functionality. Go to the feature flag UI at http://localhost:8080/feature/ and enable the productCatalogFailure feature flag. This will cause the product catalog service to return an error for a specific product ID and respond correctly to all other product IDs. Note the increase in error rate captured by AppDynamics Cloud. In AppDynamics, you can set up health rules and alerts to respond quickly to such situations.

 

OTel and AppDynamics Cloud

The OpenTelemetry Community Demo application is a valuable and safe tool for learning about OpenTelemetry and instrumentation best practices. In this blog, we showed how to configure the demo app to send telemetry data to AppDynamics Cloud. We also explored some key AppDynamics Cloud features such as the flow map, APM metrics, and an observed increase in error rates via a fault-injection scenario.

Interested in trying this workflow to learn more about OpenTelemetry? Clone and deploy the opentelemetry-demo repo and send its telemetry data to AppDynamics Cloud and request a demo.

Saurabh is currently working as Senior Engineering Manager, Quality (Application and Cloud Monitoring) at AppDynamics. He has over 15 years of experience and enjoys building, coaching and growing engineering teams to deliver high-quality customer experiences. He is passionate about learning new technologies, and his current interests are around Observability, OpenTelemetry™, Kubernetes® and all things Cloud Native.