Using OTel auto-instrumentation/agents – OpenTelemetry in Kubernetes
Maybe you heard about Opentelemetry, Kubernetes, or Opentelemetry in Kubernetes, and you don’t know what it is, or you want to learn more? This article will discuss how the OpenTelemetry (OTel) new collector feature simplifies workloads deployment on Kubernetes.
Opentelemetry is an open-source project hosted by the Cloud Native Computing Foundation (CNCF) that provides a standard way to generate telemetry data. OpenTelemetry bridges the gap between service providers and users alike when collecting data from a diverse set of systems. This helps providers and users gain a broad picture of the application’s performance and the underlying reasons for the result.
As the adoption of Cloud-native ecosystems continues to grow, the complexities of distributed systems and microservices will also expand, resulting in the need for a simple and scalable way to deploy workloads on Kubernetes.
One of the most tedious processes when deploying an observability solution is instrumentation. Presently, there are two approaches to instrument an application: manual/explicit and automatic instrumentation.
Manual/explicit: Developers can use pre-built instrumentation libraries or Opentelemetry APIs to instrument the application’s source code.
Automatic: the entire instrumentation process is on automation without any code modification and recompilation
The dawn of Opentelemetry changes a lot when automating instrumentation. What used to be a proprietary technology delivered by several Application Performance Monitoring (APM) or observability vendors, users can enjoy vendor-neutral instrumentation as an open-source technology.
Regardless of the possibility of using Opentelemtry as an open-source technology, deploying Opentelemtry auto-instrumentation at scale or validating proof of value on Kubernetes can still be a major problem. It’s hard to ignore the increasing number of immutable containers in modern applications. This necessitates that adding instrumentation to existing images requires rebuilding containers, resulting in a tedious and costly project. Using Opentelemtry Operator’s new features on Kubernetes solves this problem.
Instrumentation CR in OTel operator used in OpenTelemetry in Kubernetes
The new auto-instrumentation agent, OpenTelemetry 0.38.0, introduced a power-packed feature known as Instrumentation custom resource (CR). This feature defines the configuration for OpenTelemtry SDK. Instrumentation CR presence in the cluster and namespace of workload annotation enables SDK instrumentation in a simplified manner.
kubectl apply -f - <<EOF apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: my-instrumentation spec: exporter: endpoint: http://otel-collector:4317 propagators: - tracecontext - baggage - b3 sampler: type: parentbased_traceidratio argument: "0.25" java: image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest nodejs: image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest python: image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest EOF.
Presently, the instrumentation feature is supported across different languages such as Java, NodeJS, and Python once the workload or namespace is presented with the following annotation for different languages.
- instrumentation.opentelemetry.io/inject-java: “true” — for Java
- instrumentation.opentelemetry.io/inject-nodejs: “true” — for NodeJS
- instrumentation.opentelemetry.io/inject-python: “true” — for Python
From here, the operator can introduce OpenTelemetry auto-instrumentation libraries into the application container and then configure the instrumentation so the auto-instrumentation agent can export data to an endpoint in the instrumentation CR.
Java example with Spring Pet clinic
Looking at this from another perspective, let’s deploy the Spring Petclinic Java application with Opentelemetry collector in mind. The application will be instrumented and data communicated to an OpenTelemetry collector.
First, let’s create the following deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: spring-petclinic
spec:
selector:
matchLabels:
app: spring-petclinic
replicas: 1
template:
metadata:
labels:
app: spring-petclinic
annotations:
sidecar.opentelemetry.io/inject: "true"
instrumentation.opentelemetry.io/inject-java: "true"
spec:
containers:
- name: app
image: ghcr.io/pavolloffay/spring-petclinic:latest
and apply the instrumentation annotation:
kubectl patch deployment.apps/spring-petclinic -p '{"spec": {"template": {"metadata": {"annotations": {"instrumentation.opentelemetry.io/inject-java": "true"}}}}}'
Something unique happens when the instrumentation annotation is applied. The spring-petclinic pod restarts its operation and the newly started pod will be instrumented with OpenTelemtry Java auto-instrumentation. OTLP reports the telemetry data to the collector at http://otel-collector:4317.
We have attached the following code snippet to guide you through the deployment process.
kubectl apply -f - <<EOF apiVersion: opentelemetry.io/v1alpha1 kind: OpenTelemetryCollector metadata: name: otel spec: config: | receivers: otlp: protocols: grpc: HTTP: processors: exporters: logging: service: pipelines: traces: receivers: [otlp] processors: [] exporters: [logging] EOF
If you look closely, you can see that we can port-forward the application HTTP port via kubectl port-forward deployment.apps/spring-pet clinic 8080:8080 and review the application on the WEB. The OpenTelemetry collector can view the data spans via kubectl logs deployment.apps/otel-collector.
Application Instrumentation can be a complicated process depending on the language applied. For instance, when we look at Java, the auto-instrumentation is called Java agent, and it does bytecode manipulation. This procedure injects instrumentation points to unique code paths. After that, the points create telemetry data when the final code is executed.
How is the injection logic implemented?
The process might sound tricky, but let me explain using Java agent, although similar standards for other runtimes. The OpenTelemetry operator implements a special feature called mutating admission webhook. This feature is invoked during the creation of the Pod object, and during the process, the webhook modifies the Pod object to introduce auto-instrumentation libraries into the application container.
The Javaagent is introduced into the application container through an init container that copies that Javaagent into a volume mounted to the application container. Configuring the SDK is done by introducing environment variables.
Into the application container.
The final step is to configure the JVM to utilize the auto-instrumentation agent. The Javaagent can do this by configuring the environment variable to utilize the Javaagent.
Conclusion – OpenTelemetry in Kubernetes
In conclusion, we expect to see continued investment in Kubernetes, especially in workload deployment. OpenTelemetry auto-instrumentation on Kubernetes can simplify workload deployment with the operator pattern. There is no need to change the application container, resulting in a high value for a scalable way to transmit telemetry solutions. Presently, only Java, Python, and NodeJS runtimes are supported; hopefully, other languages will continue to gain momentum.
