Setup with OpenTelemetry

🛑Alpha

OpenTelemetry support is currently in alpha and may undergo significant changes.

This guide shows how to integrate Gentrace with existing OpenTelemetry pipelines.

Prerequisites

• An existing OpenTelemetry deployment in your application
• A Gentrace API key (available from your Gentrace dashboard)

Core concepts and architecture

OpenTelemetry is an observability framework for instrumenting, generating, collecting, and exporting telemetry data, including spans, metrics, and logs. OpenTelemetry is built around three core components:

• Instrumentation: Language SDKs and API calls that create and manage spans (units of work in your application), either through automated or manual instrumentations.
• Collection and processing: The OpenTelemetry Collector (or in-process processors like Sampler) batches, enriches, and filters spans.
• Exporters: Modules that send spans to one or more backends via protocols like OTLP/HTTP.

Gentrace supports span ingestion in accordance with the OpenTelemetry generative AI specification. Spans can be annotated with attributes (key/value metadata) and events (time-stamped messages). Context propagation (via Baggage) links spans across services. Because the OTEL collector supports multiple exporters, you can fan out identical traces to Gentrace alongside other systems by simply adding a Gentrace OTLP exporter.

To connect your OpenTelemetry pipeline to Gentrace, you'll need to:

1. Configure an OTLP/HTTP exporter to send data to Gentrace's endpoint
2. Add semantic conventions for LLM interactions either through automated or manual instrumentations
3. Use Gentrace-specific attributes to enable advanced features

Configuring your OpenTelemetry SDK

Before sending spans to Gentrace, you need to configure your OpenTelemetry SDK with an OTLP exporter. Choose the example that matches your environment:

TypeScript

ts
import { NodeSDK } from '@opentelemetry/sdk-node';
import { resourceFromAttributes } from '@opentelemetry/resources';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { ATTR_SERVICE_NAME } from '@opentelemetry/semantic-conventions';
import { GentraceSpanProcessor } from 'gentrace';
const traceExporter = new OTLPTraceExporter({
  url: 'https://gentrace.ai/api/otel/v1/traces',
  headers: {
    Authorization: `Bearer ${GENTRACE_API_KEY}`,
  },
});
const sdk = new NodeSDK({
  resource: resourceFromAttributes({
    [ATTR_SERVICE_NAME]: 'your-service-name',
  }),
  traceExporter,
  spanProcessors: [
    new GentraceSpanProcessor(),
    new SimpleSpanProcessor(traceExporter),
  ],
});
sdk.start();

Python

python
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry import trace
import os
# In virtually all cases, you should use https://gentrace.ai/api as the base URL
GENTRACE_BASE_URL = os.environ.get('GENTRACE_BASE_URL', 'https://gentrace.ai/api')
GENTRACE_API_KEY = os.environ['GENTRACE_API_KEY']
resource = Resource.create({
    "service.name": "my-gentrace-app"
})
provider = TracerProvider(resource=resource)
trace.set_tracer_provider(provider)
exporter = OTLPSpanExporter(
    endpoint=f"{GENTRACE_BASE_URL}/otel/v1/traces",
    headers={
        "Authorization": f"Bearer {GENTRACE_API_KEY}"
    },
)
processor = SimpleSpanProcessor(exporter)
provider.add_span_processor(processor)
print("OpenTelemetry SDK started – spans will be sent to Gentrace.")

The NodeSDK setup above is key for Gentrace:

• It configures the OTLP Exporter to send traces to Gentrace.
• It includes the GentraceSpanProcessor, which simplifies the process of converting baggage to span attributes:
  • This processor looks for an OpenTelemetry Baggage item named gentrace.sample.
  • If gentrace.sample is "true", the processor adds a gentrace.sample="true" attribute to the span.
  • This attribute signals to Gentrace that the span should be selected for processing.

For more details on gentrace.sample and other attributes, see the Traces API Reference.

The next sections detail how gentrace.sample is set and how spans are sent.

Automatic baggage handling with Gentrace SDK helpers

The Gentrace SDK aims to simplify common tracing scenarios. When you use the built-in helper functions like interaction() and traced(), they automatically manage the OpenTelemetry Baggage for you. Specifically, they ensure the gentrace.sample baggage item is set to "true" for the operations they trace.

Here's a conceptual illustration using interaction():

TypeScript

typescript
import OpenAI from 'openai';
import { interaction } from 'gentrace';
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
async function getOpenAIChatCompletion(prompt: string) {
  // ✅ At this point, the active span has the "gentrace.sample" attribute set
  const response = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: prompt }],
  });
  return response.choices[0].message.content;
}
const tracedOpenAICall = interaction(
  '<pipeline uuid>',
  getOpenAIChatCompletion
);
async function main() {
  // The traced invocation sets the baggage attribute on the span
  const result = await tracedOpenAICall("What is the capital of France?");
  console.log("AI Response:", result);
}

Python

python
import os
from openai import OpenAI
from gentrace import interaction, init
OPENAI_API_KEY = os.environ["OPENAI_API_KEY"]
GENTRACE_API_KEY = os.environ["GENTRACE_API_KEY"]
GENTRACE_PIPELINE_ID = os.environ["GENTRACE_PIPELINE_ID"]
init(
    api_key=GENTRACE_API_KEY,
    # Optional for self-hosted deployments: base_url=os.environ.get("GENTRACE_BASE_URL", "https://gentrace.ai/api")
)
client = OpenAI(api_key=OPENAI_API_KEY)
@interaction(pipeline_id=GENTRACE_PIPELINE_ID)
async def query_ai(query: str) -> str | None:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": query}]
    )
    return response.choices[0].message.content

Because these SDK helpers handle this for you, if you are primarily using them for your Gentrace-related tracing, you generally do not need to manually manipulate OpenTelemetry Baggage for the gentrace.sample item for those specific operations.

Manual span decoration

If you're not using the Gentrace SDK helpers (interaction, test, etc.) you can still send rich LLM traces by adding these attributes and events to your spans:

TypeScript

ts
span.setAttribute('gentrace.pipeline_id', '<pipeline-id>'); // Associate with a Gentrace Pipeline UUID
span.addEvent('gentrace.fn.args', { args: JSON.stringify([inputs]) }); // Capture function inputs
span.addEvent('gentrace.fn.output', { output: JSON.stringify(result) }); // Capture function outputs

Python

python
import json
span.set_attribute('gentrace.pipeline_id', '<pipeline-id>') # Associate with a Gentrace Pipeline UUID
span.add_event('gentrace.fn.args', { 'args': json.dumps([inputs]) }) # Capture function inputs
span.add_event('gentrace.fn.output', { 'output': json.dumps(result) }) # Capture function outputs

Serialization Requirements

When capturing function inputs and outputs:

• TypeScript: Ensure objects are properly serialized with JSON.stringify(). Handle circular references in objects to prevent serialization errors.
• Python: Pydantic objects must be converted to dictionaries (using .model_dump() or .dict() depending on version) before serialization with json.dumps().

Improper serialization can lead to data loss or errors in your traces.

For semantic LLM context, we recommend emitting OTEL Gen AI events/attributes as described in the Traces API reference.

For spans to be considered by Gentrace sampling mechanisms (either in-process or via a collector), they need to have the gentrace.sample="true" attribute. As highlighted in the "Automatic baggage handling" section, Gentrace SDK helper functions like interaction() and traced() manage this for the spans they create.

If you are manually instrumenting other parts of your code (i.e., not using Gentrace SDK helpers for those specific operations) and need those spans to be selected by Gentrace, you are responsible for ensuring the gentrace.sample="true" attribute is present on those spans. The recommended way to achieve this for consistent propagation across any child spans is to set a gentrace.sample item with the value "true" in OpenTelemetry Baggage. The GentraceSpanProcessor (configured in your NodeSDK setup, as shown earlier) will then automatically detect this baggage item and convert it into the required gentrace.sample="true" span attribute for you.

Sending spans to Gentrace

You have two main methods to control which spans get sent to Gentrace:

1. In-Process Sampling using a custom OTEL Sampler in your application.
2. Collector-Based Filtering using the OpenTelemetry Collector to filter spans.

1. In-process sampling with a custom sampler

Endpoint Configuration: Ensure your OTLP exporter is pointed at Gentrace's OTEL API (https://gentrace.ai/api/otel/v1/traces) or to your Collector's OTLP endpoint (e.g. http://collector.example.com:4318).

To implement in-process sampling, you can use a custom sampler. The Gentrace SDK exports a GentraceSampler that you can use directly:

TypeScript

typescript
import { GentraceSampler } from 'gentrace';
import { NodeSDK } from '@opentelemetry/sdk-node';
new NodeSDK({
  sampler: new GentraceSampler(), 
  // ... other NodeSDK options (resource, exporters, etc.)
}).start();

Python

python
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from gentrace import GentraceSampler
resource = Resource.create(
  attributes={
    "service.name": "my-gentrace-app"
  }
)
tracer_provider = TracerProvider(
  resource=resource, 
  sampler=GentraceSampler()
  # ... other provider options
)

This GentraceSampler is designed to check for the gentrace.sample attribute (propagated via Baggage by the GentraceSpanProcessor or Gentrace SDK helpers) and sample accordingly.

If you do not have the gentrace package installed, or if you prefer to implement custom logic, you can create your own sampler by implementing the Sampler interface from @opentelemetry/sdk-trace-base. You would typically check for the gentrace.sample attribute within your custom shouldSample method. Refer to the official OpenTelemetry documentation on Samplers for more details on custom sampler implementation. For reference, a simplified example of such a custom sampler is shown below:

2. Collector-based filtering

Endpoint Configuration: Your application should send spans to your Collector's OTLP receiver (e.g. http://collector.example.com:4318), and the Collector will export only filtered spans to Gentrace. In the Collector config below, otlphttp/gentrace still points at Gentrace's OTEL API.

Alternatively, if you're using the OpenTelemetry Collector, you can add Gentrace as an exporter and configure filtering directly in the collector. This approach is useful if you want to manage sampling and routing logic outside your application code.

First, add Gentrace as an OTLP/HTTP exporter in your collector configuration file (e.g., otel-collector-config.yaml):

yaml
exporters:
  otlphttp/gentrace:
    endpoint: https://gentrace.ai/api/otel/v1/traces
    headers:
      Authorization: "Bearer ${env:GENTRACE_API_KEY}"

Make sure to replace GENTRACE_API_KEY with your actual API key.

Then, define a pipeline that uses this exporter. For a simple setup sending all received traces to Gentrace:

yaml
service:
  pipelines:
    traces: # Default traces pipeline
      receivers: [otlp] # Assumes an OTLP receiver is configured
      processors: [batch] # Batch processor is recommended for production
      exporters: [otlphttp/gentrace]

Filtering only Gentrace spans with the Collector

If you are already sending traces to another backend (e.g., Datadog, Jaeger) and want to send only certain traces to Gentrace, you can use a filter processor. The filter/gentrace_sample processor below keeps only spans where the attribute gentrace.sample equals "true".

yaml
processors:
  filter/gentrace_sample:
    error_mode: ignore
    traces:
      span:
        # For filter processor, it's often inverted to drop:
        # Drop spans that DO NOT have gentrace.sample = "true"
        - 'attributes["gentrace.sample"] != "true"'
service:
  pipelines:
    traces/other_backend: # Pipeline for your existing backend
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/other_backend] # Replace with your other exporter, e.g., datadog
    traces/gentrace: # Dedicated pipeline for Gentrace
      receivers: [otlp]
      processors: [batch, filter/gentrace_sample]
      exporters: [otlphttp/gentrace]

In this configuration:

• Spans are received via OTLP.
• One pipeline (traces/other_backend) sends all traces to your primary observability backend.
• Another pipeline (traces/gentrace) uses the filter/gentrace_sample processor. This processor will drop spans unless the gentrace.sample attribute is present and set to "true".
• The Gentrace SDK automatically sets the gentrace.sample attribute (via Baggage, then converted by GentraceSpanProcessor) on spans created by its helper functions (interaction, test, etc.).
• If you are not using the Gentrace SDK's helpers and are manually instrumenting your OpenTelemetry client, you'll need to ensure the gentrace.sample attribute is set to "true" on the spans you intend for Gentrace.

Note: Regardless of which method you choose, you must propagate the gentrace.sample baggage entry into your spans so that sampling and filtering work correctly. The GentraceSpanProcessor from the above code example is critical.

ts
import { GentraceSpanProcessor } from 'gentrace';
new NodeSDK({
  // ... existing NodeSDK options (resource, exporters, sampler, etc.)
  spanProcessors: [
    new GentraceSpanProcessor(),
    // ... other processors
  ],
}).start();

Next steps

• See the Traces API Reference for details on expected span attributes and events