OpenTelemetry Auto-Instrumentation Configuration ¶

When you enable auto-instrumentation in your application the following OpenTelemetry configuration will become available to your application as environment variables:

Variable	Example Value
`OTEL_SERVICE_NAME`	`my-application`
`OTEL_EXPORTER_OTLP_ENDPOINT`	`http://opentelemetry-collector.nais-system:4317`
`OTEL_EXPORTER_OTLP_PROTOCOL`	`grpc`
`OTEL_EXPORTER_OTLP_INSECURE`	`true`
`OTEL_PROPAGATORS`	`tracecontext,baggage`
`OTEL_TRACES_SAMPLER`	`parentbased_always_on`
`OTEL_LOGS_EXPORTER`	`none`
`OTEL_RESOURCE_ATTRIBUTES_POD_NAME`	`my-application-777787df6d-pw9mq`
`OTEL_RESOURCE_ATTRIBUTES_NODE_NAME`	`gke-node-abc123`
`OTEL_RESOURCE_ATTRIBUTES`	`service.name=my-application,service.namespace=my-team,k8s.container.name=my-application,...`

Do not hardcode these values in your application or try to overwrite them in your nais.yaml. OpenTelemetry SDKs and auto-instrumentation libraries will automatically pick up these environment variables and use them to configure the SDK depending on where your application is running.

Extra resource attributes ¶

OpenTelemetry Resource Attributes are key-value pairs that describe the application and its environment. These attributes are attached to all telemetry data produced by the application and must adhere to the OpenTelemetry Semantic Conventions.

Most applications will not need to set extra resource attributes, and only some attributes can be set by the application using the OTEL_RESOURCE_ATTRIBUTES environment variable.

We have compile a list of attributes that can be set by the application and we advise you to consult the specification for more information about how they should be used:

Attribute	Type	Description	Example Value	Stability
`deployment.environment.name`	string	The version string of the service API or implementation. The format is not defined by these conventions.	`q1`; `p`	`experimental`

Sanitizing sensitive data ¶

While the OpenTelemetry SDKs and agends does a best effort to not include any user data, we have added extra protection by redacting some patterns from some trace span attribtes. The following attributes are scrubbed for personal idification numbers (fødselsnummer):

url.path
url.full
url.original
url.query
http.url
http.target
http.route
db.statement
messaging.kafka.message.key

We advise you to look over the data collected by your application and ensure that no sensitive data is being stored and contact the nais-team if you have any questions or concerns.

Logs auto-instrumentation ¶

You can enable instrumenting logs using the OpenTelemetry Auto-Instrumentation by setting the OTEL_LOGS_EXPORTER environment variable to otlp. This will intercept all logs produced by the application and send them to the OpenTelemetry Collector.

shell

spec:
  env:
    - name: OTEL_LOGS_EXPORTER
      value: otlp

Enabling logging for the OpenTelemetry Auto-Instrumentation will send all logs to the OpenTelemetry Collector including logs from other libraries and frameworks such as log4j, logback, and slf4j. This should not be enabled if you are using Team Logs.

Agent Versions ¶

The OpenTelemetry Agent is used to automatically instrument your application. The agent is responsible for collecting telemetry data and sending it to the OpenTelemetry Collector.

Runtime	Agent Version	SDK Version
`java`	2.22.0	1.55.0
`nodejs`	0.67.0	0.208.0
`python`	0.51b0	1.30.0
`dotnet`	1.12.0

Java Agent ¶

The OpenTelemetry Java Agent is a Java agent that automatically instruments your Java application. The agent is responsible for collecting telemetry data and sending it to the OpenTelemetry Collector.

It is attached to your JVM automatically at startup using the JAVA_TOOL_OPTIONS environment variable.

Supported libraries, frameworks, application servers, and JVMs ¶

The OpenTelemetry Java Agent supports many popular libraries, frameworks, application servers, and JVMs. A full list of supported libraries and frameworks can be found on the open-telemetry/opentelemetry-java-instrumentation repository.

OpenTelemetry Instrumentation for Java Supported Libraries and Frameworks

Advanced Configuration ¶

When using the OpenTelemetry Java SDK and Agent (auto-instrumentation), the following additional environment variables are available:

Variable	Description	Example Value
`OTEL_JAVAAGENT_LOGGING`	Controles log output from the Java Agent itself, valid values are `none`, `simple`, and `application`	`simple`
`OTEL_JAVAAGENT_EXCLUDE_CLASSES`	Suppresses all instrumentation for specific classes, format is "my.package.MyClass,my.package2.*"	`my.package.MyClass,my.package2.*`
`OTEL_INSTRUMENTATION_SPRING_BOOT_ACTUATOR_AUTOCONFIGURE_ENABLED`	Enables or disables the Spring Boot Actuator auto-configuration instrumentation	`false`
`OTEL_INSTRUMENTATION_MICROMETER_ENABLED`	Enables or disables the Micrometer instrumentation	`false`
`OTEL_INSTRUMENTATION_COMMON_EXPERIMENTAL_CONTROLLER_TELEMETRY_ENABLED`	Enables or disables controller span instrumentation	`false`
`OTEL_INSTRUMENTATION_COMMON_EXPERIMENTAL_VIEW_TELEMETRY_ENABLED`	Enables or disables view span instrumentation	`false`

More OpenTelemetry Configuration ¶

A full list of environment variables that can be used to configure the OpenTelemetry SDK can be found here:

OTLP is the OpenTelemetry Protocol, and is the protocol used to send telemetry data to Prometheus, Grafana Tempo, and Grafana Loki.

Destinations ¶

If you need to override where telemetry data is stored, you can do so with the following configuration in your nais.yaml:

yaml

spec:
  observability:
    telemetry:
      destinations:
        - grafana-lgtm

The following destinations are available:

grafana-lgtm

Default destination for each environment can be found in the environments overview.