Skip to main content
Version: 4.7.66

Traces

🛑Alpha

OpenTelemetry support is currently in alpha and may undergo significant changes.

This document details how to submit trace data to Gentrace via the OpenTelemetry Protocol (OTLP) and outlines the expected span attributes and events for effective integration.

OTLP endpoint​

Gentrace ingests OpenTelemetry data via HTTP.

  • Endpoint URL: /api/otel/v1/traces (Relative to your Gentrace instance base URL)
  • Method: POST
  • Format: OTLP/HTTP (protobuf)
Recommendation

You should use either the OpenTelemetry Collector or the Gentrace SDK (which handle OTLP export internally) to submit trace data. These methods provide more robust features, batching, and easier configuration. You can learn more about setting up OTLP in the OpenTelemetry Setup Guide.

Key span attributes​

Spans sent to Gentrace should include the following attributes.

Gentrace specific attributes​

These attributes link spans to specific Gentrace entities or control ingestion behavior:

  • gentrace.pipeline_id: (String) The ID of the Gentrace Pipeline this span belongs to. This is automatically set by the interaction() SDK wrapper.
  • gentrace.experiment_id: (String) The ID of the Gentrace Experiment this span is part of. This is automatically set when using the experiment() function with objects like eval() or evalDataset() within the SDK.
  • gentrace.test_case_id: (String) (Optional) The ID of a persisted Gentrace Test Case. Automatically set by evalDataset() if the underlying data provides an ID that corresponds to a TestCase in Gentrace.
  • gentrace.test_case_name: (String) The descriptive name of the test case. Automatically set by eval() or evalDataset().
  • gentrace.sample: (String: "true") (Optional) This attribute indicates that the span is intended for Gentrace. The Gentrace SDK uses OpenTelemetry Baggage to automatically set this attribute on relevant spans. Users can then filter spans using a sampler within the SDK or in the OTEL collector.

Semantic convention attributes (generative AI)​

Gentrace utilizes key OpenTelemetry semantic convention attributes for generative AI at the span level to understand the broader context of LLM interactions. For full details on all GenAI span attributes, see the OpenTelemetry GenAI Span Conventions.

Key span-level attributes include:

  • gen_ai.system: (String) The AI system used (e.g., "openai", "anthropic").
  • gen_ai.request.model: (String) The specific model requested (e.g., "gpt-4o").
  • gen_ai.operation.name: (String) The specific operation performed by the LLM (e.g., "chat", "completion", "embedding"). (Note: The server also looks for operation.name for backward compatibility with older Traceloop conventions).

For detailed prompt and completion content, including individual messages and tool calls, Gentrace follows the OpenTelemetry specification which recommends using events. See the "Semantic Convention Events (Generative AI)" section below.

Traceloop Attribute Compatibility

The server can also parse span attributes from Traceloop semantic conventions (e.g., llm.request.type, or flattened prompt/completion attributes like llm.prompts.0.role). However, adhering to the official OpenTelemetry GenAI event-based conventions is recommended.

Other standard attributes​

  • service.name: (String) Identifies the service generating the span (e.g., "my-chat-service").
  • span.name: (String) The name of the span, often the function or operation name being traced.

Key span events​

Events provide detailed insights into specific points within a span's lifecycle. Each event has a name and can have its own set of attributes.

Gentrace SDK events​

These events are automatically emitted by the Gentrace SDK wrappers (traced, interaction, eval):

  • Event Name: gentrace.fn.args
    • Description: Marks the invocation of a function traced by the SDK.
    • Attribute: args (String - JSON array): A stringified JSON array of the arguments passed to the function.
  • Event Name: gentrace.fn.output
    • Description: Marks the successful completion of a function traced by the SDK.
    • Attribute: output (String - JSON): A stringified JSON representation of the function's return value.

Semantic convention events (generative AI)​

For capturing detailed prompt and completion data, including individual messages, roles, content, and tool calls, OpenTelemetry's semantic conventions for Generative AI specify the use of events. Gentrace fully supports this event-based approach.

Refer to the official OpenTelemetry GenAI Events Specification for the complete and authoritative documentation on event names and their attributes.

Key event patterns include:

  • Event Name Pattern: gen_ai.message (or system-specific variants like gen_ai.openai.message)
    • Description: Represents a single message in a prompt or completion sequence.
    • Key Attributes on the Event:
      • message.role: (String) The role of the message (e.g., "user", "assistant", "system", "tool").
      • message.content: (String) The textual content of the message.
      • message.name: (String, Optional) If the role is "tool", the name of the tool.
      • message.tool_calls: (String, Optional, JSON) If the role is "assistant" and it makes tool calls, this contains an array of tool call objects (e.g., [{ "id": "call_abc", "type": "function", "function": { "name": "tool_name", "arguments": "{...}" } }]).
  • Event Name: gen_ai.choice
    • Description: Represents a choice or completion generated by an LLM, particularly when multiple completions are generated.
    • Key Attributes on the Event:
      • choice.index: (Integer) The index of this choice.
      • choice.content: (String) The content of this specific choice/completion.
      • choice.role: (String, Optional) The role associated with this choice (e.g., "assistant").
      • choice.finish_reason: (String, Optional) The reason the LLM stopped generating tokens for this choice.
      • choice.tool_calls: (String, Optional, JSON) Tool calls associated with this specific choice.

By sending these events on your LLM interaction spans, Gentrace can accurately reconstruct the full conversation flow and tool usage.

Standard OpenTelemetry events​

Gentrace also processes standard OpenTelemetry events, particularly for error reporting:

  • Event Name: exception
    • Description: Indicates an error or exception occurred during the span's execution. See OpenTelemetry exception recording for more details.
    • Attributes:
      • exception.type: (String) The type or class name of the exception (e.g., Error, TypeError).
      • exception.message: (String) The error message.
      • exception.stacktrace: (String) The stack trace associated with the exception.

This documentation provides a concise overview. For more in-depth information on OpenTelemetry concepts, refer to the official OpenTelemetry Specification.

See also​