Skip to main content

​
Overview

This guide covers how to integrate Gentrace with complex OpenTelemetry setups. If you are looking for a basic Gentrace setup, start with our quickstart.

​
Core Concepts

OpenTelemetry uses three main components:
​
Instrumentation
component
Language SDKs and APIs that create and manage spans (units of work in your application)
​
Processing
component
The OpenTelemetry Collector or in-process processors that batch, enrich, and filter spans
​
Exporters
component
Modules that send spans to backends via protocols like OTLP/HTTP
Gentrace supports span ingestion following the OpenTelemetry generative AI specification. Spans can include:
  • Attributes: Key/value metadata
  • Events: Time-stamped messages
  • Context propagation: Links spans across services via Baggage
In this guide, we cover how to setup the following span processing infrastructure to send only relevant spans to Gentrace:

​
Steps

To send your traces that use LLMs to Gentrace via OpenTelemetry, we need to:
1

Instrument your application

Include Gentrace and gen_ai attributes and events in your spans.
2

Export spans to Gentrace

Set up span infrastructure to filter and export spans, so only relevant spans are sent to Gentrace.
3

View your traces in Gentrace

Head to your Gentrace dashboard to see your traces.

​
Instrument your application

​
Create the spans

Use the following SDK helpers to automatically add Gentrace attributes and events to your spans: Alternatively, add Gentrace attributes and events manually: 
span.setAttribute('gentrace.pipeline_id', '<pipeline-id>');
span.addEvent('gentrace.fn.args', { args: JSON.stringify([inputs]) });
span.addEvent('gentrace.fn.output', {
  output: JSON.stringify(result),
});
For a comprehensive list of Gentrace-specific attributes and events, including semantic conventions for generative AI, see the OpenTelemetry API Reference. This guide covers all supported span attributes, events, and their expected formats.
Serialization Requirements - TypeScript: Use JSON.stringify() and handle circular references - Python: Convert Pydantic objects to dictionaries before json.dumps()

​
Set the baggage for future filtering

For existing OpenTelemetry setups, spans must be filtered before exporting to Gentrace to avoid rate limiting. To facilitate this, add the baggage item gentrace.sample=true to relevant spans, and then filter in your exporter on the basis of gentrace.sample.
This is done automatically for you when using the interaction() SDK helper.
span.setBaggage('gentrace.sample', 'true');

​
Test span marking

When running experiments, the SDK automatically adds gentrace.in_experiment=true to the baggage to mark test spans. This prevents production evaluations from running on test data. You should set this manually in your experiment script: 
span.setBaggage('gentrace.sample', 'true');
span.setBaggage('gentrace.in_experiment', 'true');
Baggage is automatically propagated to child spans recursively.

​
Export spans to Gentrace

First, choose one of two methods:

With the OpenTelemetry Collector

Use OpenTelemetry Collector to filter and route spans

With In-Process Sampling

Use a custom sampler in your application to control which spans are sent

​
With the OpenTelemetry Collector

With the OpenTelemetry Collector, setup the following span processing infrastructure to send only relevant spans to Gentrace:

​
1. Map baggage to span attributes

The collector does not receive baggage items, as they are only held in-memory. Use a span processor to map gentrace.sample and gentrace.in_experiment from the baggage to the span. For convenience, we provide a span processor that does this for you: 
import { GentraceSpanProcessor } from 'gentrace';

const gentraceSpanProcessor = new GentraceSpanProcessor();

const sdk = new NodeSDK({
  // ...
  spanProcessors: [gentraceSpanProcessor],
  // ...
});

​
2. Add Gentrace exporter to the collector configuration

exporters:
  otlphttp/gentrace:
    endpoint: https://gentrace.ai/api/otel/v1/traces
    headers:
      Authorization: 'Bearer ${env:GENTRACE_API_KEY}'

​
3. Configure filtering to only send Gentrace-specific spans

processors:
  filter/gentrace_sample:
    error_mode: ignore
    traces:
      span:
        # Drop spans that don't have gentrace.sample = "true"
        - 'attributes["gentrace.sample"] != "true"'

​
4. Set up pipelines

service:
  pipelines:
    traces/gentrace:
      receivers: [otlp]
      processors: [batch, filter/gentrace_sample]
      exporters: [otlphttp/gentrace]
Point your application’s OTLP exporter to your Collector (e.g., http://collector.example.com:4318), which will then forward filtered spans to Gentrace’s OTLP endpoint at /api/otel/v1/traces

​
Complete example collector configuration

Open this configuration in OTELbin to validate and visualize your Collector configuration

​
With In-Process Sampling

With the OpenTelemetry SDK, setup the following span processing infrastructure to send only relevant spans to Gentrace:

​
1. Sample spans based on baggage

Before setting up your exporter, sample spans based on gentrace.sample=true. For simplicity, we supply the GentraceSampler, which is an in-process sampler that filters spans based on OpenTelemetry baggage. 
import { NodeSDK } from '@opentelemetry/sdk-node';
import { GentraceSampler } from 'gentrace';

const sdk = new NodeSDK({
  // ...
  sampler: new GentraceSampler(),
  // ...
});

​
2. Export spans to Gentrace

Add a Gentrace exporter to your OpenTelemetry SDK configuration: 
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base';

const sdk = new NodeSDK({
  // ...
  sampler: new GentraceSampler(),
  new SimpleSpanProcessor(
    new OTLPTraceExporter({
      url: 'https://gentrace.ai/api/otel/v1/traces',
      headers: {
        Authorization: `Bearer ${GENTRACE_API_KEY}`,
      },
    })
  )
  // ...
});

​
Troubleshooting

If spans aren’t appearing in Gentrace:
  1. Verify filtering: Ensure spans have the gentrace.sample=true attribute (set by SDK helpers or manually)
  2. Check collector configuration: Confirm the filter processor isn’t dropping your spans
  3. Validate API key: Ensure your GENTRACE_API_KEY is correctly set in the collector
  4. Monitor ingestion:
    Visit https://gentrace.ai/s/otel-metrics to view ingestion metrics including distributions of accepted, rejected, and buffered spans
  5. Test collector connectivity: Use the OpenTelemetry Collector’s built-in logging to debug connection issues

​
Next Steps