Metadata

You can attach run-level metadata with our SDKs to attach related information to your generative output.

Usage

You can add run-level context using both our simple and advanced SDKs.

Supported keys

Some keys only work on certain types of AI generations. For example, user IDs can only be associated to an entire generation, whereas metadata can be assigned directly on particular steps within a generation. Visit the detailed page for each context type to learn more about its semantics.

User IDsuserId

Associate user IDs from your own service with particular AI generations

ThreadingpreviousRunId

Associate a series of AI generations together (e.g. chat bots)

Render formatrender

Render a key in the outputs payload with a custom format. Currently, we can only render HTML at this time.

Other metadata

Attach arbitrary key/value information to your observed AI data.

Simple SDK

Every simple invocation allows you to specify a gentrace parameter, where you can supply specific context key/value pairs. Here's an example from our OpenAI simple plugin.

TypeScript

typescript
import { init } from '@gentrace/core';
import { OpenAI } from '@gentrace/openai';
 
init({
  apiKey: process.env.GENTRACE_API_KEY,
});
 
const openai = new OpenAI({
  apiKey: process.env.OPENAI_KEY,
});
 
await openai.chat.completions.create({
  messages: [
    {
      role: 'user',
      contentTemplate:
        'Hello! My name is {{ name }}. Write a brief essay about Maine.',
      contentInputs: { name: 'Vivek' },
    },
  ],
  model: 'gpt-3.5-turbo',
  pipelineSlug: 'simple-pipeline',
  stream: true,
 
  // Context is specified with the `gentrace` key. In this case, the user ID will be
  // associated with this generation.
  gentrace: {
    userId: 'TWFuIGlzIGRpc3Rpbmd1aXNoZW',
  },
});

Python

python
import openai
import gentrace
import os
openai = gentrace.OpenAI(api_key=os.getenv("OPENAI_KEY"))
openai.chat.completions.create(
    messages=[
        {
            "role": "user",
            "contentTemplate": "Hello! My name is {{ name }}. Write a brief essay about Maine.",
            "contentInputs": {"name": "Vivek"},
        }
    ],
    model="gpt-3.5-turbo",
    pipeline_slug="example-completion",
    # Context is specified with the `gentrace` key. In this case, the user ID will be
    # associated with this generation.
    gentrace={
        "userId": "TWFuIGlzIGRpc3Rpbmd1aXNoZW",
    },
)

Advanced SDK

You can specify context when you initialize a PipelineRun instance with the pipeline.start() function.

TypeScript

typescript
import { Pipeline } from '@gentrace/core';
import { initPlugin } from '@gentrace/openai';
 
const plugin = await initPlugin({
  apiKey: process.env.OPENAI_KEY,
});
 
const pipeline = new Pipeline({
  slug: 'advanced-pipeline',
  plugins: {
    openai: plugin,
  },
});
 
// Context is specified directly as an object. This context is associated with all
// steps in the generative pipeline.
const runner = pipeline.start({
  userId: 'TWFuIGlzIGRpc3Rpbmd1aXNoZW',
});
 
const openai = runner.openai;
 
const chatCompletionResponse = await openai.chat.completions.create({
  messages: [{ role: 'user', content: 'Hello!' }],
  model: 'gpt-3.5-turbo',
  stream: true,
});

Python

python
import gentrace
PIPELINE_SLUG = "compose"
pipeline = gentrace.Pipeline(
    PIPELINE_SLUG,
    openai_config={
        "api_key": process.env.OPENAI_KEY,
    },
)
runner = pipeline.start({
  "userId": "TWFuIGlzIGRpc3Rpbmd1aXNoZW",
})
openai = runner.get_openai()
result = openai.chat.completions.create(
  messages=[{ "role": "user", "content": "Hello!" }],
  model="gpt-3.5-turbo",
  stream=True
)

You can also associate context with any individual step with the gentrace parameter.

TypeScript

typescript
import { Pipeline } from '@gentrace/core';
import { initPlugin } from '@gentrace/openai';
 
const plugin = await initPlugin({
  apiKey: process.env.OPENAI_KEY,
});
 
const pipeline = new Pipeline({
  slug: 'advanced-pipeline',
  plugins: {
    openai: plugin,
  },
});
 
const runner = pipeline.start();
 
const openai = runner.openai;
 
const chatCompletionResponse = await openai.chat.completions.create({
  messages: [{ role: 'user', content: 'Hello!' }],
  model: 'gpt-3.5-turbo',
  stream: true,
 
  // Context is applied directly to this AI step
  gentrace: {
    userId: 'TWFuIGlzIGRpc3Rpbmd1aXNoZW',
  },
});

Python

python
import gentrace
gentrace.init(
    api_key=os.getenv("GENTRACE_API_KEY"),
)
openai = gentrace.OpenAI(api_key=os.getenv("OPENAI_KEY"))
openai.api_key = os.getenv("OPENAI_KEY")
result = openai.chat.completions.create(
    pipeline_slug="testing-chat-completion-value",
    messages=[
        {
            "role": "user",
            "contentTemplate": "Hello {{ name }}! What's your name?",
            "contentInputs": {"name": "John"},
        }
    ],
    model="gpt-3.5-turbo",
    gentrace={
        "userId": "TWFuIGlzIGRpc3Rpbmd1aXNoZW",
    },
)

Certain step methods like measure() and checkpoint() require that context is explicitly passed in a context key.

TypeScript

typescript
import { Pipeline } from '@gentrace/core';
import { initPlugin } from '@gentrace/openai';
 
const plugin = await initPlugin({
  apiKey: process.env.OPENAI_KEY,
});
 
const pipeline = new Pipeline({
  slug: 'advanced-pipeline',
  plugins: {
    openai: plugin,
  },
});
 
const runner = pipeline.start();
 
const outputs = await runner.measure(
  (pageDescription) => {
    // ... Omitted logic for creating HTML from provided inputs
    return {
      koalaHtml: htmlOutput,
    };
  },
  ['Create HTML for a site about koalas'],
  {
    context: {
      render: {
        type: 'html',
        key: 'koalaHtml',
      },
    },
  },
);
 
await runner.submit();

Python

python
import gentrace
def generate_html(description):
  # ... Generate HTML
  return {
    "koalaHtml": htmlOutput
  }
PIPELINE_SLUG = "compose"
pipeline = gentrace.Pipeline(
    PIPELINE_SLUG,
    openai_config={
        "api_key": process.env.OPENAI_KEY,
    },
)
runner = pipeline.start()
output = runner.measure(
  generate_html,
  description="Create HTML for a site about koalas",
  step_info={
    "context": {
      "render": {
        "type": "html",
        "key": "koalaHtml"
      }
    }
  },
)
runner.submit()

OpenAI automatic capture (cost, speed)

Some metadata attributes are automatically captured by our OpenAI SDK. These include:

• cost in US$
• speed in milliseconds

TypeScript

typescript
const runner = pipeline.start();
 
const openai = runner.openai;
 
const chatCompletionResponse = await openai.chat.completions.create({
  messages: [{ role: 'user', content: 'Hello!' }],
  model: 'gpt-3.5-turbo',
  stream: true,
});
 
// ✅ By invoking the create() function, the cost and speed of that generation
// are automatically captured.

Python

python
runner = pipeline.start()
openai = runner.get_openai()
result = openai.chat.completions.create(
  messages=[{ "role": "user", "content": "Hello!" }],
  model="gpt-3.5-turbo",
  stream=True
)
# ✅ By invoking the create() function, the cost and speed of that generation
# are automatically captured.

Streaming costs

OpenAI does not return usage information for streaming completions. When this happens, we estimate the cost by tokenizing the input/output information with tiktoken.

Streaming tool call costs are estimates

OpenAI does not disclose how tool call definitions are tokenized internally. In this case, we estimate costs with the following heuristic. Note: Your actual OpenAI costs may deviate slightly from this heuristic.

TypeScript

typescript
// Constants for estimating tool call costs
const ONE_TOOL_PROMPT_TOKENS = 19;
const MULTI_TOOL_INITIAL_JUMP = 35;
const MULTI_TOOL_PROGRESSIVE_JUMP = 12;
function estimateToolCallTokens(
  toolCallDefinitionTokenCount: number,
  numToolCalls: number,
): number {
  if (toolCallCount === 1) {
    return toolCallDefinitionTokenCount + ONE_TOOL_PROMPT_TOKENS;
  } else {
    const offset =
      ONE_TOOL_PROMPT_TOKENS +
      (toolCallCount - 2) * MULTI_TOOL_PROGRESSIVE_JUMP +
      MULI_TOOL_INITIAL_JUMP;
    return toolCallDefinitionTokenCount + offset;
  }
}

Python

python
# Constants for estimating tool call costs
ONE_TOOL_PROMPT_TOKENS = 19
MULTI_TOOL_INITIAL_JUMP = 35
MULTI_TOOL_PROGRESSIVE_JUMP = 12
def estimate_tool_call_tokens(tool_call_definition_token_count, num_tool_calls):
    if num_tool_calls == 1:
        return tool_call_definition_token_count + ONE_TOOL_PROMPT_TOKENS
    else:
        offset = ONE_TOOL_PROMPT_TOKENS + (num_tool_calls - 2) * \
          MULTI_TOOL_PROGRESSIVE_JUMP + MULTI_TOOL_INITIAL_JUMP
        return tool_call_definition_token_count + offset

Manual cost capture

To capture costs from fine-tuned OpenAI models and other LLM providers, you can manually specify the cost of your model's generation by providing the following structure in your submitted output:

Data structure

json
{
  "cost": {
    "type": "dollars", // or "cents"
    "value": 1.0 // 100
  }
}

Simple

To capture costs with the simple test result submission flow, specify the cost in the output object.

TypeScript

typescript
const tcs = await getTestCases(pipelineSlug);
const outputs = await Promise.all(
  tcs.map((testCase) => {
    // ... Omit logic for creating generative output
    return {
      generativeOutput: '... ',
      cost: {
        type: 'cents',
        value: 100,
      },
    };
  }),
);
 
await submitTestResult(pipelineSlug, tcs, outputs);

Python

python
test_cases = gentrace.get_test_cases(pipeline_slug=PIPELINE_SLUG)
outputs = []
for test_case in test_cases:
    inputs = test_case["inputs"]
    # Omit logic for creating generative output
    outputs.push({
      "generativeOutput": "...",
      "cost": {
          "type": "cents",
          "value": 100
      }
    })
response = gentrace.submit_test_result(PIPELINE_SLUG, test_cases, outputs)

Advanced

To capture costs with the advanced SDK runner, specify the cost using a custom measure() function.

TypeScript

typescript
const outputs = await runner.measure(
  (pageDescription) => {
    // ... Omit logic for generating HTML from page description
    return {
      html: htmlOutput,
      cost: {
        type: 'cents',
        value: 100,
      },
    };
  },
  ['Create HTML for a site about koalas'],
);
 
await runner.submit();

Python

python
def generate_html(description):
    # ... Omit logic for generating HTML from page description
    return {
        "koalaHtml": htmlOutput,
        "cost": {
            "type": "cents",
            "value": 100
        }
    }
outputs = runner.measure(
    generate_html,
    description="Create HTML for a site about koalas",
)
runner.submit()