How to observe OpenTelemetry demo app in Cloud Native Application Observability

Since its launch in May 2019, the OpenTelemetry (OTel) Project has become one of the most popular open source projects since Kubernetes®. For newcomers to the observability domain, OpenTelemetry™ provides a standard way to collect telemetry data (metric, logs and traces) from software applications and send it to one or more backends to analyze application performance. The backends can be open source (Jaeger or Zipkin, for example), commercial (such as Cloud Native Application Observability or AppDynamics APM) or both.

To enable faster adoption and showcase instrumentation best practices, the OTel community has built a demo application, OpenTelemetry Community Demo. In this blog, I’ll show how to configure the OpenTelemetry demo to send telemetry data to Cloud Native Application Observability for further analysis.

Backstory: Cloud Native Application Observability and OpenTelemetry demo

Cloud Native Application Observability, our new software-as-a-service (SaaS) solution, provides full-stack observability of cloud native applications and natively supports OpenTelemetry-based ingestion.

OpenTelemetry Community Demo is a simulated version of an eCommerce store selling astronomy equipment. The app consists of 14+ microservices communicating with each other via HTTP or GRPC. The microservices are built using a variety of programming languages (Java, Javascript, C#, etc.) and instrumented using OpenTelemetry (auto, manual or both). The diagram below shows the data flow and programming languages used.

Image credit: OpenTelemetry Demo contributors.

In addition to the microservices shown here, the demo app also comes with supporting components such as OpenTelemetry Collector, Grafana, Prometheus and Jaeger to export and visualize traces, metrics and so on. OpenTelemetry Collector can be configured to send telemetry data to multiple backends simultaneously. The diagram below shows the OTel demo with supporting components, as well as a dotted line to Cloud Native Application Observability, which we will configure in the next section.

Sending telemetry data to Cloud Native Application Observability

1. 1. 1. 1. Using the steps described in the Open Telemetry demo Docker deployment documentation, deploy the demo app on your local machine.
         2. Contact your AppDynamics account representative to set up an AppDynamics tenant for your company. The tenant will have a URL format similar to https://<your-company-name>.observe.appdynamics.com and will be the central location where you’ll see all telemetry data from your applications.
         3. For this Docker setup, we’ll use the credentials generated for Kubernetes and APM monitoring. To generate credentials, see the steps highlighted in the AppDynamics UI below.
            
         4. We will use the following key-values:
            a. clientId:<value>
            b. clientSecret:<value>
            c. endpoint:<value>
            d. tokenUrl:<value>
         5. As described below, update the file src/otelcollector/otelcol-config-extras.yml in your cloned repo. Make sure there are no yaml validation errors by opening this file in an IDE with yaml support (VSCode etc.). The contents of this file get merged with src/otelcollector/otelcol-config.yml at runtime to create the consolidated OpenTelemetry Collector configuration.

            extensions: 
            # AppD CNAO requires enabling oath2client extension for sending data to it 
            
              oauth2client/appd: 
                client_id: <from step 4a above> 
                client_secret: <from step 4b above > 
                token_url: <from step 4d above> 
                
            exporters: 
              otlphttp/appd:    
                traces_endpoint: <from step 4c above>/v1beta/trace 
                metrics_endpoint: <from step 4c above>/v1beta/metrics        
                auth: 
                  authenticator: oauth2client/appd 
             
            processors: 
              batch: 
              resource: 
                attributes: 
                - key: service.namespace 
                  action: upsert    
                  value: otel-demo 
             
            service: 
              extensions: [oauth2client/appd] 
              pipelines: 
                traces: 
                  receivers: [otlp] 
                  processors: [batch, resource] 
                  exporters: [logging, otlp, otlphttp/appd] 
                metrics: 
                  receivers: [otlp] 
                  processors: [batch, resource] 
                  exporters: [logging, prometheus, otlphttp/appd]

             

         6. Stop the docker containers (‘docker compose down’)  and then start them  (‘ docker compose up -d’). Wait a few minutes and confirm that you can access the OpenTelemetry demo app UI at http://localhost:8080/. Next, log in to your Cloud Native Application Observability Tenant URL and click the “Service Topology” shortcut button. You’ll then see a service flow map that shows various microservices and the interactions between them.
            
         7. Click the List view to display key APM metrics such as Avg Response Time, Calls/min and Errors/min.
            
         8. An observability platform should be able to detect an increase in error rates of the microservices it is monitoring. Fortunately, the OpenTelemetry demo has an error injection capability to test this functionality. Go to the feature flag UI at http://localhost:8080/feature/ and enable the productCatalogFailure feature flag. This will cause the product catalog service to return an error for a specific product ID and respond correctly to all other product IDs. Note the increase in error rate captured by Cloud Native Application Observability. In AppDynamics, you can set up health rules and alerts to respond quickly to such situations.

OTel and Cloud Native Application Observability

The OpenTelemetry Community Demo application is a valuable and safe tool for learning about OpenTelemetry and instrumentation best practices. In this blog, we showed how to configure the demo app to send telemetry data to Cloud Native Application Observability. We also explored some key Cloud Native Application Observability features such as the flow map, APM metrics, and an observed increase in error rates via a fault-injection scenario.

Interested in trying this workflow to learn more about OpenTelemetry? Clone and deploy the opentelemetry-demo repo and send its telemetry data to Cloud Native Application Observability and request a demo.

Saurabh Deshpande

Saurabh is currently working as Senior Engineering Manager, Quality (Application and Cloud Monitoring) at AppDynamics. He has over 15 years of experience and enjoys building, coaching and growing engineering teams to deliver high-quality customer experiences. He is passionate about learning new technologies, and his current interests are around Observability, OpenTelemetry™, Kubernetes® and all things Cloud Native.