Skip to main content
Version: 4.7.8

Threading

A thread is a back-and-forth exchange between your generative AI pipeline and an external actor (typically an end user). A prime example is the conversation that occurs between ChatGPT and a user. Typically, a thread is composed of multiple AI generations.

Within our data model, we represent threads as a singly-linked list of pipeline runs. You can specify that a run belongs to a thread by specifying the prior pipeline run while generating the next run.

Example

Let's say you have a chat feature for your end user. The feature initially performs a chat completion request with our SDK.

typescript
import { init } from "@gentrace/core";
import { OpenAI } from "@gentrace/openai";
 
init({
apiKey: process.env.GENTRACE_API_KEY,
});
 
const openai = new OpenAI({
apiKey: process.env.OPENAI_KEY,
});
 
// This would normally be populated from an HTTP request
const endUserQuestion = "Hello OpenAI! How are you doing today?";
 
const chatCompletionResponse = await openai.chat.completions.create({
messages: [
{
role: "user",
content: endUserQuestion
},
],
model: "gpt-3.5-turbo",
pipelineSlug: "introduction",
});
 
const runId = chatCompletionResponse.pipelineRunId;
 
// We have omitted logic:
// - to store this run in your desired persistent storage (e.g. PostgRES
// - to return the chat response to the user

Once the request is performed and you receive a Gentrace run ID, store the initial pipeline run ID in persistent storage (e.g. database).

At a later point, your end user decides to respond to the supplied AI generation with a follow up question.

typescript
import { OpenAI } from "@gentrace/openai";
 
const openai = new OpenAI({
apiKey: process.env.OPENAI_KEY,
});
 
// This would normally be populated from an HTTP request
const endUserQuestion = "Great to hear! What's the capital of Maine?";
 
const previousRunId = ...; // TODO: pull prior run ID from DB
const priorMessages = ...; // TODO: pull prior messages from DB
 
const chatCompletionResponse = await openai.chat.completions.create({
messages: [
...priorMessages,
{
role: "user",
content: endUserQuestion
},
],
model: "gpt-3.5-turbo",
pipelineSlug: "introduction",
gentrace: {
// By specifying the previous run ID, we associate this generation to a thread in Gentrace
previousRunId,
}
});
 
const runId = chatCompletionResponse.pipelineRunId;
 
// We have omitted logic:
// - to store this run in your desired persistent storage (e.g. PostgRES
// - to return the chat response to the user

Once the generation completes, Gentrace will associate that generation with the previously-specified run.

UI

You can view threads in the Observe → Runs view. Runs that belong to the same thread are rolled up into a single row.

View threads in run page

While in the detailed view for the thread, navigate through the individual runs with the left/right arrow keys.

Navigate thread in detail page

You can also navigate through individual runs in the timeline view.

Navigate thread in timeline page

Limitations

caution

This feature is available for our observability features only.

caution

The previousRunId parameter cannot be specified on a single step within a multi-step pipeline.

To demonstrate why, let's consider the below incorrect code, which makes two calls to OpenAI in a single generation.

typescript
import { Pipeline } from "@gentrace/core";
import { initPlugin } from "@gentrace/openai";
 
const plugin = await initPlugin({
apiKey: process.env.OPENAI_KEY,
});
 
const pipeline = new Pipeline({
slug: "advanced-pipeline",
plugins: {
openai: plugin,
},
});
 
const runner = pipeline.start();
 
const openai = runner.openai;
 
const chatCompletionResponse = await openai.chat.completions.create({
messages: [{ role: "user", content: "Hello! My name is Vivek." }],
model: "gpt-3.5-turbo",
// 🛑 This is not correct since it's assigned to a single step
gentrace: {
previousRunId: "67be57c5-fd7d-4bd8-9c6c-17991a4d689f",
}
});
 
const chatCompletionResponse2 = await openai.chat.completions.create({
messages: [{ role: "user", content: "Hello! My name is Doug." }],
model: "gpt-3.5-turbo",
});
 
await runner.submit()
 

In Gentrace, a previousRunId is always associated with a single Gentrace run. In the case of our advanced SDK, multiple steps within a single runner are considered part of a single run.

caution

Threading supports only singly-linked runs. We currently do not support branching where multiple generations reference the same run. If two runs are submitted with the same previous run ID (previousRunId), one run submission will be rejected with a 400 status code.

Types

previousRunId: string (UUID)

Future work

  • Thread evaluation
  • Visual thread comparison
  • Aggregate statistics (e.g. average run latency)
  • Thread metadata

If you're interested in shaping this future work, please reach out over email.