> ## Documentation Index
> Fetch the complete documentation index at: https://next.gentrace.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# OpenTelemetry guide

> This advanced guide covers how to integrate Gentrace with complex OpenTelemetry setups.

## Overview

This guide covers how to integrate Gentrace with complex OpenTelemetry setups.

If you are looking for a basic Gentrace setup, start with our [quickstart](/getting-started/quickstart).

## Core Concepts

OpenTelemetry uses three main components:

<ResponseField name="Instrumentation" type="component">
  Language SDKs and APIs that create and manage spans (units of work
  in your application)
</ResponseField>

<ResponseField name="Processing" type="component">
  The OpenTelemetry Collector or in-process processors that batch,
  enrich, and filter spans
</ResponseField>

<ResponseField name="Exporters" type="component">
  Modules that send spans to backends via protocols like OTLP/HTTP
</ResponseField>

Gentrace supports span ingestion following the [OpenTelemetry generative AI specification](https://opentelemetry.io/docs/specs/semconv/gen-ai/). Spans can include:

* **Attributes**: Key/value metadata
* **Events**: Time-stamped messages
* **Context propagation**: Links spans across services via [Baggage](https://opentelemetry.io/docs/concepts/signals/baggage/)

In this guide, we cover how to setup the following span processing infrastructure to send only relevant spans to Gentrace:

## Steps

To send your traces that use LLMs to Gentrace via OpenTelemetry, we need to:

<Steps>
  <Step title="Instrument your application">
    Include Gentrace and gen\_ai attributes and events in your spans.
  </Step>

  <Step title="Export spans to Gentrace">
    Set up span infrastructure to filter and export spans, so only
    relevant spans are sent to Gentrace.
  </Step>

  <Step title="View your traces in Gentrace">
    Head to your [Gentrace dashboard](https://gentrace.ai/t) to see
    your traces.
  </Step>
</Steps>

## Instrument your application

### Create the spans

Use the following SDK helpers to automatically add Gentrace attributes and events to your spans:

* [interaction()](../tracing/interactions) - for sending agent interactions to Gentrace
* [traced()](../tracing/traced) - for tracing functions

Alternatively, add Gentrace attributes and events manually:

<CodeGroup dropdown>
  ```typescript theme={null}
  span.setAttribute('gentrace.pipeline_id', '<pipeline-id>');
  span.addEvent('gentrace.fn.args', { args: JSON.stringify([inputs]) });
  span.addEvent('gentrace.fn.output', {
    output: JSON.stringify(result),
  });
  ```

  ```python theme={null}
  import json
  span.set_attribute('gentrace.pipeline_id', '<pipeline-id>')
  span.add_event('gentrace.fn.args', { 'args': json.dumps([inputs]) })
  span.add_event('gentrace.fn.output', { 'output': json.dumps(result) })
  ```
</CodeGroup>

<Tip>
  For a comprehensive list of Gentrace-specific attributes and events,
  including semantic conventions for generative AI, see the
  [OpenTelemetry API Reference](./opentelemetry-api). This guide
  covers all supported span attributes, events, and their expected
  formats.
</Tip>

<Warning>
  **Serialization Requirements** - **TypeScript**: Use
  `JSON.stringify()` and handle circular references - **Python**:
  Convert Pydantic objects to dictionaries before `json.dumps()`
</Warning>

### Set the baggage for future filtering

For existing OpenTelemetry setups, spans must be filtered *before* exporting to Gentrace to avoid rate limiting.

To facilitate this, add the baggage item `gentrace.sample=true` to relevant spans, and then filter in your exporter on the basis of `gentrace.sample`.

<Tip>
  This is done automatically for you when using the
  [interaction()](../tracing/interactions) SDK helper.
</Tip>

<CodeGroup dropdown>
  ```typescript theme={null}
  span.setBaggage('gentrace.sample', 'true');
  ```

  ```python theme={null}
  span.set_baggage('gentrace.sample', 'true')
  ```
</CodeGroup>

#### Test span marking

When running experiments, the SDK automatically adds `gentrace.in_experiment=true` to the baggage to mark test spans.
This prevents production evaluations from running on test data.
You should set this manually in your experiment script:

<CodeGroup dropdown>
  ```typescript theme={null}
  span.setBaggage('gentrace.sample', 'true');
  span.setBaggage('gentrace.in_experiment', 'true');
  ```

  ```python theme={null}
  context = otel_context.get_current()
  context = otel_baggage.set_baggage("gentrace.sample", "true", context=context)
  context = otel_baggage.set_baggage("gentrace.in_experiment", "true", context=context)
  token = otel_context.attach(context)
  ```
</CodeGroup>

Baggage is automatically propagated to child spans recursively.

## Export spans to Gentrace

First, choose one of two methods:

<CardGroup cols={2}>
  <Card title="With the OpenTelemetry Collector" icon="server" href="#with-the-opentelemetry-collector">
    Use OpenTelemetry Collector to filter and route spans
  </Card>

  <Card title="With In-Process Sampling" icon="filter" href="#with-in-process-sampling">
    Use a custom sampler in your application to control which spans
    are sent
  </Card>
</CardGroup>

### With the OpenTelemetry Collector

With the OpenTelemetry Collector, setup the following span processing infrastructure to send only relevant spans to Gentrace:

<Frame>
  ```mermaid theme={null}
  graph TB
      A["📦 Incoming Span"]
      B["⚙️ GentraceSpanProcessor<br/>Checks for baggage"]
      A --> B
      B --> C{"Has gentrace.sample=true<br/>in baggage?"}
      C -->|"✅ YES"| D["✅ Add as span attribute"]
      C -->|"❌ NO"| E["❌ No attribute added"]
      D --> F["🔍 OpenTelemetry Collector<br/>filters by attribute"]
      E --> F
      F --> G{"Has gentrace.sample<br/>attribute?"}
      G -->|"✅ YES"| H["✅ Forward to Gentrace"]
      G -->|"❌ NO"| I["🗑️ Discarded"]
  ```
</Frame>

#### 1. Map baggage to span attributes

The collector does not receive baggage items, as they are only held in-memory.

Use a span processor to map `gentrace.sample` and `gentrace.in_experiment` from the baggage to the span.

For convenience, we provide a span processor that does this for you:

<CodeGroup dropdown>
  ```typescript theme={null}
  import { GentraceSpanProcessor } from 'gentrace';

  const gentraceSpanProcessor = new GentraceSpanProcessor();

  const sdk = new NodeSDK({
    // ...
    spanProcessors: [gentraceSpanProcessor],
    // ...
  });
  ```

  ```python theme={null}
  from gentrace import GentraceSpanProcessor
  from opentelemetry.sdk.trace import TracerProvider

  provider = TracerProvider()
  provider.add_span_processor(GentraceSpanProcessor())
  ```
</CodeGroup>

#### 2. Add Gentrace exporter to the collector configuration

```yaml theme={null}
exporters:
  otlphttp/gentrace:
    endpoint: https://gentrace.ai/api/otel/v1/traces
    headers:
      Authorization: 'Bearer ${env:GENTRACE_API_KEY}'
```

#### 3. Configure filtering to only send Gentrace-specific spans

```yaml theme={null}
processors:
  filter/gentrace_sample:
    error_mode: ignore
    traces:
      span:
        # Drop spans that don't have gentrace.sample = "true"
        - 'attributes["gentrace.sample"] != "true"'
```

#### 4. Set up pipelines

```yaml theme={null}
service:
  pipelines:
    traces/gentrace:
      receivers: [otlp]
      processors: [batch, filter/gentrace_sample]
      exporters: [otlphttp/gentrace]
```

<Note>
  Point your application's OTLP exporter to your Collector (e.g.,
  `http://collector.example.com:4318`), which will then forward
  filtered spans to Gentrace's OTLP endpoint at `/api/otel/v1/traces`
</Note>

#### Complete example collector configuration

<Expandable title="Full Collector Configuration">
  ```yaml theme={null}
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

  processors:
    batch:
      timeout: 1s
      send_batch_size: 1024

    filter/gentrace_sample:
      error_mode: ignore
      traces:
        span:
          # Drop spans that don't have gentrace.sample = "true"
          - 'attributes["gentrace.sample"] != "true"'

  exporters:
    otlphttp/gentrace:
      endpoint: https://gentrace.ai/api/otel/v1/traces
      headers:
        Authorization: 'Bearer ${env:GENTRACE_API_KEY}'

    # Example: if you also want to send to other backends
    otlphttp/other_backend:
      endpoint: https://your-other-backend.com/v1/traces

  service:
    pipelines:
      # Pipeline for all traces to other backends
      traces/other_backend:
        receivers: [otlp]
        processors: [batch]
        exporters: [otlphttp/other_backend]

      # Pipeline for Gentrace-specific traces
      traces/gentrace:
        receivers: [otlp]
        processors: [batch, filter/gentrace_sample]
        exporters: [otlphttp/gentrace]
  ```

  Save this as `otel-collector-config.yaml` and run the OpenTelemetry Collector with your Gentrace API key:

  ```bash theme={null}
  # If you installed the collector as a binary
  GENTRACE_API_KEY=your-api-key-here otelcol --config=otel-collector-config.yaml

  # Or if using Docker
  docker run -v $(pwd)/otel-collector-config.yaml:/etc/otel-collector-config.yaml \
    -e GENTRACE_API_KEY=your-api-key-here \
    otel/opentelemetry-collector:latest \
    --config=/etc/otel-collector-config.yaml
  ```

  <Note>
    Replace `your-api-key-here` with your actual Gentrace API key from your [dashboard](https://gentrace.ai/s/api-keys).
  </Note>

  See the [OpenTelemetry Collector installation guide](https://opentelemetry.io/docs/collector/installation/) for platform-specific instructions.
</Expandable>

<Tip>
  [Open this configuration in
  OTELbin](https://www.otelbin.io/#config=receivers%3A*N__otlp%3A*N____protocols%3A*N______grpc%3A*N________endpoint%3A_0.0.0.0%3A4317*N______http%3A*N________endpoint%3A_0.0.0.0%3A4318*N*Nprocessors%3A*N__batch%3A*N____timeout%3A_1s*N____send*_batch*_size%3A_1024*N__*N__filter%2Fgentrace*_sample%3A*N____error*_mode%3A_ignore*N____traces%3A*N______span%3A*N________*H_Drop_spans_that_don%27t_have_gentrace.sample_%3D_%22true%22*N________-_%27attributes%5B%22gentrace.sample%22%5D_!%3D_%22true%22%27*N*Nexporters%3A*N__otlphttp%2Fgentrace%3A*N____endpoint%3A_https%3A%2F%2Fgentrace.ai%2Fapi%2Fotel%2Fv1%2Ftraces*N____headers%3A*N______Authorization%3A_%27Bearer_%24%7Benv%3AGENTRACE*_API*_KEY%7D%27*N__*N__*H_Example%3A_if_you_also_want_to_send_to_other_backends*N__otlphttp%2Fother*_backend%3A*N____endpoint%3A_https%3A%2F%2Fyour-other-backend.com%2Fv1%2Ftraces*N*Nservice%3A*N__pipelines%3A*N____*H_Pipeline_for_all_traces_to_other_backends*N____traces%2Fother*_backend%3A*N______receivers%3A_%5Botlp%5D*N______processors%3A_%5Bbatch%5D*N______exporters%3A_%5Botlphttp%2Fother*_backend%5D*N____*N____*H_Pipeline_for_Gentrace-specific_traces*N____traces%2Fgentrace%3A*N______receivers%3A_%5Botlp%5D*N______processors%3A_%5Bbatch%2C_filter%2Fgentrace*_sample%5D*N______exporters%3A_%5Botlphttp%2Fgentrace%5D%7E)
  to validate and visualize your Collector configuration
</Tip>

### With In-Process Sampling

With the OpenTelemetry SDK, setup the following span processing infrastructure to send only relevant spans to Gentrace:

<Frame>
  ```mermaid theme={null}
  graph TB
      A["📦 Incoming Span"]
      B["⚙️ GentraceSampler<br/>Samples spans based on baggage"]
      A --> B
      B --> C{"Has gentrace.sample=true<br/>in baggage?"}
      C -->|"✅ YES"| D["✅ Forward to Gentrace"]
      C -->|"❌ NO"| E["❌ Discarded"]
  ```
</Frame>

#### 1. Sample spans based on baggage

Before setting up your exporter, sample spans based on `gentrace.sample=true`.

For simplicity, we supply the `GentraceSampler`, which is an in-process sampler that filters spans based on OpenTelemetry baggage.

<CodeGroup dropdown>
  ```typescript theme={null}
  import { NodeSDK } from '@opentelemetry/sdk-node';
  import { GentraceSampler } from 'gentrace';

  const sdk = new NodeSDK({
    // ...
    sampler: new GentraceSampler(),
    // ...
  });
  ```

  ```python theme={null}
  from gentrace import GentraceSampler
  from opentelemetry.sdk.trace import TracerProvider

  provider = TracerProvider(
      # ...
      sampler=GentraceSampler()
      # ...
  )
  ```
</CodeGroup>

#### 2. Export spans to Gentrace

Add a Gentrace exporter to your OpenTelemetry SDK configuration:

<CodeGroup dropdown>
  ```typescript theme={null}
  import { NodeSDK } from '@opentelemetry/sdk-node';
  import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
  import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base';

  const sdk = new NodeSDK({
    // ...
    sampler: new GentraceSampler(),
    new SimpleSpanProcessor(
      new OTLPTraceExporter({
        url: 'https://gentrace.ai/api/otel/v1/traces',
        headers: {
          Authorization: `Bearer ${GENTRACE_API_KEY}`,
        },
      })
    )
    // ...
  });
  ```

  ```python theme={null}
  from gentrace import GentraceSampler
  from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
  from opentelemetry.sdk.trace.export import SimpleSpanProcessor
  from opentelemetry.sdk.trace import TracerProvider

  exporter = OTLPSpanExporter(
      endpoint="https://gentrace.ai/api/otel/v1/traces",
      headers={
          "Authorization": f"Bearer {GENTRACE_API_KEY}"
      },
  )

  provider = TracerProvider(
      # ...
      sampler=GentraceSampler()
      # ...
  )
  provider.add_span_processor(SimpleSpanProcessor(exporter))
  ```
</CodeGroup>

## Troubleshooting

If spans aren't appearing in Gentrace:

1. **Verify filtering**: Ensure spans have the `gentrace.sample=true` attribute (set by SDK helpers or manually)
2. **Check collector configuration**: Confirm the filter processor isn't dropping your spans
3. **Validate API key**: Ensure your `GENTRACE_API_KEY` is correctly set in the collector
4. **Monitor ingestion**:
   <Warning>
     Visit
     [https://gentrace.ai/s/otel-metrics](https://gentrace.ai/s/otel-metrics)
     to view ingestion metrics including distributions of accepted,
     rejected, and buffered spans
   </Warning>
5. **Test collector connectivity**: Use the OpenTelemetry Collector's built-in logging to debug connection issues

## Next Steps

* [OpenTelemetry API Reference](./opentelemetry-api) - Span attributes and events
* [OpenTelemetry SDK setup](./opentelemetry-api) - Detailed SDK configuration
* [interaction()](../tracing/interactions) - SDK helper for tracing functions
* [experiment()](../evaluation/experiments) - SDK helper for experiments