Gentrace Documentation

Overview

This guide covers how to integrate Gentrace with complex OpenTelemetry setups. If you are looking for a basic Gentrace setup, start with our quickstart.

Core Concepts

OpenTelemetry uses three main components:

Instrumentation

component

Language SDKs and APIs that create and manage spans (units of work in your application)

Processing

component

The OpenTelemetry Collector or in-process processors that batch, enrich, and filter spans

Exporters

component

Modules that send spans to backends via protocols like OTLP/HTTP

Gentrace supports span ingestion following the OpenTelemetry generative AI specification. Spans can include:

Attributes: Key/value metadata
Events: Time-stamped messages
Context propagation: Links spans across services via Baggage

In this guide, we cover how to setup the following span processing infrastructure to send only relevant spans to Gentrace:

Steps

To send your traces that use LLMs to Gentrace via OpenTelemetry, we need to:

Instrument your application

Include Gentrace and gen_ai attributes and events in your spans.

Export spans to Gentrace

Set up span infrastructure to filter and export spans, so only relevant spans are sent to Gentrace.

View your traces in Gentrace

Head to your Gentrace dashboard to see your traces.

Instrument your application

Create the spans

Use the following SDK helpers to automatically add Gentrace attributes and events to your spans:

interaction() - for sending agent interactions to Gentrace
traced() - for tracing functions

Alternatively, add Gentrace attributes and events manually:

span.setAttribute('gentrace.pipeline_id', '<pipeline-id>');
span.addEvent('gentrace.fn.args', { args: JSON.stringify([inputs]) });
span.addEvent('gentrace.fn.output', { output: JSON.stringify(result) });

For a comprehensive list of Gentrace-specific attributes and events, including semantic conventions for generative AI, see the OpenTelemetry API Reference. This guide covers all supported span attributes, events, and their expected formats.

Serialization Requirements

TypeScript: Use JSON.stringify() and handle circular references
Python: Convert Pydantic objects to dictionaries before json.dumps()

Set the baggage for future filtering

For existing OpenTelemetry setups, spans must be filtered before exporting to Gentrace to avoid rate limiting. To facilitate this, add the baggage item gentrace.sample=true to relevant spans, and then filter in your exporter on the basis of gentrace.sample.

This is done automatically for you when using the interaction() SDK helper.

span.setBaggage('gentrace.sample', 'true');

Baggage is automatically propagated to child spans recursively.

Export spans to Gentrace

First, choose one of two methods:

With the OpenTelemetry Collector

Use OpenTelemetry Collector to filter and route spans

With In-Process Sampling

Use a custom sampler in your application to control which spans are sent

With the OpenTelemetry Collector

With the OpenTelemetry Collector, setup the following span processing infrastructure to send only relevant spans to Gentrace:

1. Map gentrace.sample=true to the span

The collector does not receive baggage items, as they are only held in-memory. Use a span processor to map gentrace.sample from the baggage to the span. For convenience, we provide a span processor that does this for you:

import { GentraceSpanProcessor } from 'gentrace';

const gentraceSpanProcessor = new GentraceSpanProcessor();

const sdk = new NodeSDK({
  // ...
  spanProcessors: [gentraceSpanProcessor],
  // ...
});

2. Add Gentrace exporter to the collector configuration

exporters:
  otlphttp/gentrace:
    endpoint: https://gentrace.ai/api/otel/v1/traces
    headers:
      Authorization: 'Bearer ${env:GENTRACE_API_KEY}'

3. Configure filtering to only send Gentrace-specific spans

processors:
  filter/gentrace_sample:
    error_mode: ignore
    traces:
      span:
        # Drop spans that don't have gentrace.sample = "true"
        - 'attributes["gentrace.sample"] != "true"'

4. Set up pipelines

service:
  pipelines:
    traces/gentrace:
      receivers: [otlp]
      processors: [batch, filter/gentrace_sample]
      exporters: [otlphttp/gentrace]

Point your application’s OTLP exporter to your Collector (e.g., http://collector.example.com:4318), which will then forward filtered spans to Gentrace’s OTLP endpoint at /api/otel/v1/traces

Complete example collector configuration

Show Full Collector Configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024
  
  filter/gentrace_sample:
    error_mode: ignore
    traces:
      span:
        # Drop spans that don't have gentrace.sample = "true"
        - 'attributes["gentrace.sample"] != "true"'

exporters:
  otlphttp/gentrace:
    endpoint: https://gentrace.ai/api/otel/v1/traces
    headers:
      Authorization: 'Bearer ${env:GENTRACE_API_KEY}'
  
  # Example: if you also want to send to other backends
  otlphttp/other_backend:
    endpoint: https://your-other-backend.com/v1/traces

service:
  pipelines:
    # Pipeline for all traces to other backends
    traces/other_backend:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/other_backend]
    
    # Pipeline for Gentrace-specific traces
    traces/gentrace:
      receivers: [otlp]
      processors: [batch, filter/gentrace_sample]
      exporters: [otlphttp/gentrace]

Save this as otel-collector-config.yaml and run the OpenTelemetry Collector with your Gentrace API key:

# If you installed the collector as a binary
GENTRACE_API_KEY=your-api-key-here otelcol --config=otel-collector-config.yaml

# Or if using Docker
docker run -v $(pwd)/otel-collector-config.yaml:/etc/otel-collector-config.yaml \
  -e GENTRACE_API_KEY=your-api-key-here \
  otel/opentelemetry-collector:latest \
  --config=/etc/otel-collector-config.yaml

Replace your-api-key-here with your actual Gentrace API key from your dashboard.

See the OpenTelemetry Collector installation guide for platform-specific instructions.

Open this configuration in OTELbin to validate and visualize your Collector configuration

With In-Process Sampling

With the OpenTelemetry SDK, setup the following span processing infrastructure to send only relevant spans to Gentrace:

1. Sample spans based on baggage

Before setting up your exporter, sample spans based on gentrace.sample=true. For simplicity, we supply the GentraceSampler, which is an in-process sampler that filters spans based on OpenTelemetry baggage.

import { NodeSDK } from '@opentelemetry/sdk-node';
import { GentraceSampler } from 'gentrace';

const sdk = new NodeSDK({
  // ...
  sampler: new GentraceSampler(),
  // ...
});

2. Export spans to Gentrace

Add a Gentrace exporter to your OpenTelemetry SDK configuration:

import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base';

const sdk = new NodeSDK({
  // ...
  sampler: new GentraceSampler(),
  new SimpleSpanProcessor(
    new OTLPTraceExporter({
      url: 'https://gentrace.ai/api/otel/v1/traces',
      headers: {
        Authorization: `Bearer ${GENTRACE_API_KEY}`,
      },
    })
  )
  // ...
});

Troubleshooting

If spans aren’t appearing in Gentrace:

Verify filtering: Ensure spans have the gentrace.sample=true attribute (set by SDK helpers or manually)
Check collector configuration: Confirm the filter processor isn’t dropping your spans
Validate API key: Ensure your GENTRACE_API_KEY is correctly set in the collector
Monitor ingestion:
Visit https://gentrace.ai/s/otel-metrics to view ingestion metrics including distributions of accepted, rejected, and buffered spans
Test collector connectivity: Use the OpenTelemetry Collector’s built-in logging to debug connection issues

Next Steps

OpenTelemetry API Reference - Span attributes and events
OpenTelemetry SDK setup - Detailed SDK configuration
interaction() - SDK helper for tracing functions
experiment() - SDK helper for experiments

Getting started

Error analysis

Tracing

Evaluation

Integrations

Administration

Reference

SDK Entities

OpenTelemetry guide

Overview

Core Concepts

Steps

Instrument your application

Create the spans

Set the baggage for future filtering

Export spans to Gentrace

With the OpenTelemetry Collector

With In-Process Sampling

With the OpenTelemetry Collector

1. Map gentrace.sample=true to the span

2. Add Gentrace exporter to the collector configuration

3. Configure filtering to only send Gentrace-specific spans

4. Set up pipelines

Complete example collector configuration

With In-Process Sampling

1. Sample spans based on baggage

2. Export spans to Gentrace

Troubleshooting

Next Steps

Getting started

Error analysis

Tracing

Evaluation

Integrations

Administration

Reference

SDK Entities

​Overview

​Core Concepts

​Steps

​Instrument your application

​Create the spans

​Set the baggage for future filtering

​Export spans to Gentrace

With the OpenTelemetry Collector

With In-Process Sampling

​With the OpenTelemetry Collector

​1. Map gentrace.sample=true to the span

​2. Add Gentrace exporter to the collector configuration

​3. Configure filtering to only send Gentrace-specific spans

​4. Set up pipelines

​Complete example collector configuration

​With In-Process Sampling

​1. Sample spans based on baggage

​2. Export spans to Gentrace

​Troubleshooting

​Next Steps

Overview

Core Concepts

Steps

Instrument your application

Create the spans

Set the baggage for future filtering

Export spans to Gentrace

With the OpenTelemetry Collector

1. Map gentrace.sample=true to the span

2. Add Gentrace exporter to the collector configuration

3. Configure filtering to only send Gentrace-specific spans

4. Set up pipelines

Complete example collector configuration

With In-Process Sampling

1. Sample spans based on baggage

2. Export spans to Gentrace

Troubleshooting

Next Steps