Skip to main content
Version: 2.0.0

Observe quickstart (OpenAI)

This guide covers how to get started with Gentrace Observe for OpenAI. Observe monitors your logged requests and tracks speed, cost and aggregate statistics.

The other critical part of our product is Gentrace Evaluate, which automates grading your generative pipeline on a variety of dimensions. We will not cover Evaluate in this guide. Visit here to get started with Evaluate.

Installation​

This package is the simplest way to interact with our service in your application.

bash
# Execute only one, depending on your package manager
npm i @gentrace/core
yarn add @gentrace/core
pnpm i @gentrace/core

If you want to use our provider SDK handlers, you must install our associated plugin SDKs. These SDKs have a direct dependency on the officially supported SDK for their respective providers. We type match the official SDK whenever possible.

shell
# For OpenAI v4 (the new version)
npm install @gentrace/openai@v4
# For OpenAI v3 (the older version)
npm install @gentrace/openai@v3

These NPM packages will only work with Node.JS versions >= 16.16.0.

🛑Server-side only

Please only use this library on the server-side. Using it on the client-side will reveal your API key.

Simple SDK​

Requires @gentrace/openai@v4 or @gentrace/openai@v3

This section requires Gentrace's official OpenAI plugin. The plugin version matches the major version of the official OpenAI Node.JS SDK.

We designed our SDKs to mostly preserve the original interface to OpenAI's client library. You can simply insert the following lines of code before your OpenAI invocations.

typescript
import { init } from "@gentrace/core";
import { OpenAI } from "@gentrace/openai";
 
// This function globally initializes Gentrace with the appropriate 
// credentials. Constructors like OpenAI() will transparently use
// these credentials to authenticate with Gentrace.
init({
  apiKey: process.env.GENTRACE_API_KEY
});
 
const openai = new OpenAI({
  apiKey: process.env.OPENAI_KEY,
});

The OpenAI class is virtually identical to the equivalents in the official SDK.

You can then execute your OpenAI functions against the openai handle directly.

typescript
async function createEmbedding() {
  const embeddingResponse = await openai.embeddings.create({
    model: "text-embedding-ada-002",
    input: "Example input",
    // IMPORTANT: Supply a Gentrace Pipeline slug to track this invocation
    pipelineSlug: "create-test-embedding"
  });
  console.log("Pipeline run ID: ", embeddingResponse.pipelineRunId);
}
createEmbedding();
caution

You should provide a Pipeline slug as a request parameter to any method that you want to instrument. This ID associates OpenAI invocations to that identifier on our service. If you omit the slug, we will not track telemetry for that invocation.

The PipelineRun ID provided by the OpenAI create() return value is from the Gentrace SDK. Our SDK provides this for you to uniquely associate feedback with AI generated content. If you do not provide a Pipeline slug, the create() functions will not return a PipelineRun ID.

Prompt templates for the openai.completions.create() interface​

One difference between the Gentrace-instrumented SDK and the official SDK is how prompts are specified for openai.completions.create() requests.

In the official version of the SDK, you must specify the prompt as a string. In our SDK, you must specify a prompt template and it's associated inputs. We use Mustache templating with the Mustache.js library to render the prompt templates.

typescript
// ❌ Official SDK invocation
await openai.completion.create({
  ...DEFAULT_PARAMS,
  prompt: "What's the trendiest neighborhood in New York City?"
});
// ✅ Gentrace OpenAI Handle
await openai.completions.create({
  ...DEFAULT_PARAMS,
  promptTemplate: "What's {{ interestingFact }} in {{ location }}?",
  promptInputs: {
    interestingFact: "the trendiest neighborhood",
    location: "New York City"
  },
});

Note: We still allow you to specify the original prompt parameter if you want to incrementally migrate your invocations.

Consult OpenAI's Node.JS SDK documentation for more details to learn more about the original SDK.

Content templates for the openai.chat.completions.create() interface​

The other difference between the Gentrace-instrumented SDK and the official SDK is how prompts are specified for openAi.chat.completion.create() requests.

In the official version of the SDK, you specify your chat completion input as an object array with role and content key-pairs defined.

typescript
// ❌ Official OpenAI SDK invocation
const chatCompletionResponse = await openai.chat.completions.create({
  messages: [
    {
      role: "user",
      content: "Hello Vivek!"
    },
  ],
  model: "gpt-3.5-turbo"
});

In our SDK, if part of the content is dynamically generated, you should instead create contentTemplate and contentInputs key-pairs to separate the static and dynamic information, respectively. This is helpful to better display the generation in our UI and internally track version changes.

We use Mustache templating with the Mustache.js library to render the final content that is sent to OpenAI.

typescript
// ✅ Gentrace-instrumented OpenAI SDK
const chatCompletionResponse = await openai.chat.completions.create({
  messages: [
    {
      role: "user",
      contentTemplate: "Hello {{ name }}!",
      contentInputs: { name: "Vivek" },
    },
  ],
  model: "gpt-3.5-turbo",
  pipelineSlug: "testing-pipeline-id",
});

Note: We still allow you to specify the original content key-value pair in the dictionary if you want to incrementally migrate your invocations.

Consult OpenAI's Node.JS SDK documentation for more details to learn more about the original SDK.

Streaming​

We transparently wrap OpenAI's Node streaming functionality.

typescript
// Imports and initialization truncated
async function main() {
  const streamChat = await openai.chat.completions.create({
    model: 'gpt-4',
    messages: [{ role: 'user', content: 'Say this is a test' }],
    stream: true,
  });
  
  // This PipelineRun ID actually hasn't been created on the server yet.
  // It's created asynchronously after the final stream event is processed.
  console.log("Pipeline run ID: ", streamChat.pipelineRunId);
  
  for await (const part of streamChat) {
    console.log(part.choices[0]?.delta?.content || '');
  }
  
  // Stream data is coalesced and sent to our server.
}
main();
caution

We measure the total time a generation takes from the first byte received from iterating on the stream to the last event yielded from the stream.

Before sending the streamed events to Gentrace, we coalesce the streamed payload as a single payload to improve readability.

Tracking multiple invocations in one Pipeline​

With the functions shown thus far, you can only track a single OpenAI invocation per Pipeline. Check out the advanced section below to learn about our methods for tracking multiple invocations across multiple providers (e.g. Pinecone vector query + OpenAI embedding call) in a single Pipeline.

Telemetry support​

We automatically capture analytics from these OpenAI SDK methods. We plan to support other methods upon request. In these points, openai is an instance of the OpenAI class.

  • openai.completions.create()
  • openai.chat.completions.create()
  • openai.embeddings.create()

Advanced usage​

The SDKs described above are designed for creating single invocations to one provider like OpenAI or Pinecone. We also provide abstractions for chaining multiple invocations together into a single pipeline.

Creating Pipeline and PipelineRuns​

To declare a Pipeline, you must define the configuration (including API keys) for Gentrace and the services you intend to monitor.

typescript
import { init, Pipeline } from "@gentrace/core";
import { initPlugin as initOpenAIPlugin } from "@gentrace/openai";
// This function globally initializes Gentrace with the appropriate 
// credentials. Constructors like Pipeline() will transparently use
// these credentials to authenticate with Gentrace.
init({
  apiKey: process.env.GENTRACE_API_KEY
});
const openaiPlugin = await initOpenAIPlugin({
  apiKey: process.env.OPENAI_KEY,
});
const pineconePlugin = await initPineconePlugin({
  apiKey: process.env.PINECONE_API_KEY,
});
const pipeline = new Pipeline({
  slug: "searchCompanyKnowledge",
  plugins: {
    openai: openaiPlugin,
    pinecone: pineconePlugin
  },
});
Global Pipeline

We designed the Pipeline class to specify the static, global configuration of a pipeline. Then, we expect users to use this global Pipeline reference to create additional PipelineRun instances via pipeline.start(). More on that below.

To create a PipelineRun, invoke the following code. The returned runner allows you to interact with providers like OpenAI and Pinecone.

typescript
const runner = await pipeline.start();

To access a handle on a supported provider like OpenAI or Pinecone, invoke the following code.

typescript
// For OpenAI
const openAi = runner.openai;
// For Pinecone
const pinecone = runner.pinecone;

You can then access methods for these external services on the handlers. These clients are nearly API-compatible with their equivalent official SDKs. There are a few key differences we’ll get into later when we cover each provider in detail.

Submission​

Once you've invoked all the requests you need, you can submit this data to our external provider with the following code. This functionality asynchronously sends the PipelineRun data to our servers and returns a PipelineRun ID that you can send to your client.

typescript
const { pipelineRunId } = await runner.submit()

If you want to wait for the result of the submission request, you can invoke the following.

typescript
const { pipelineRunId } = await runner.submit({
  waitForServer: true
})
// This will block until the request returns

The PipelineRun ID is used to associate user feedback with the generated content. It is important to pass this ID to your client application so that you can effectively link user feedback to the corresponding AI-generated content. To facilitate this association, you can use the browser Feedback SDK.

Advanced SDK​

Requires @gentrace/openai@v4 or @gentrace/openai@v3

This section requires Gentrace's official OpenAI plugin. The plugin version matches the major version of the official OpenAI Node.JS SDK.

Our package provides a near type-match for the OpenAI Node.JS SDK. To get an instrumented version of the OpenAI SDK, simply invoke the following code.

typescript
const openaiPlugin = await initPlugin({
  apiKey: process.env.OPENAI_KEY,
});
const pipeline = new Pipeline({
  slug: "openai",
  plugins: {
    openai: openaiPlugin,
  },
});
const runner = pipeline.start();
const openai = runner.openai;
const embeddingResponse = await openai.embeddings.create({
  model: "text-embedding-ada-002",
  input: "What is Vivek's birthday?",
});

You can then invoke functions against the resulting handle that match the official SDK.

Note that in the Simple SDK, you had to specify a pipelineSlug for your invocations. If you're using the Pipeline object, the Pipeline slug is declared explicitly in the Pipeline object constructor. Similarly, the result of an invocation will not return a PipelineRun ID.

Configuration​

To configure Gentrace's Node.JS SDK with OpenAI, you must initialize a plugin using the initPlugin() method exported from every Gentrace plugin. Then, pass the same parameter object that you would pass to the OpenAI constructor as the first parameter to initPlugin().

typescript
import { init, Pipeline } from "@gentrace/core";
import { initPlugin } from "@gentrace/openai";
 
// The provided parameter object has the same types as the OpenAI constructor.
const openaiPlugin = await initPlugin({
  apiKey: process.env.OPENAI_KEY,
});
 
const pipeline = new Pipeline({
  slug: "searchCompanyKnowledge",
  // ... other configuration
  plugins: {
    openai: openaiPlugin
  }
});
 

Prompt templates​

The only difference between the interface of the Gentrace-instrumented SDK and official SDK is how prompts are specified. Consult the section about prompt templates in the OpenAI Simple SDK section earlier in this guide for more information.

Telemetry support​

We automatically capture analytics from these OpenAI SDK methods. We plan to support other methods upon request. In these points, openai is an instance of the OpenAI class.

  • openai.chat.completions.create()
  • openai.completions.create()
  • openai.embeddings.create()

Full example​

Here's a full example of a PipelineRun invocation with multiple calls to OpenAI and Pinecone (quickstart here)..

typescript
export async function generateKnowledgeResponse(input: string) {
  const runner = pipeline.start();
  
  // Near type matches for the respective clients
  const openai = runner.openai;
  const pinecone = runner.pinecone;
  
  const embeddingResponse = await openai.embeddings.create({
    model: 'text-embedding-ada-002',
    input,
  });
  
  const vectorQuery = // process embedding response
  
  // getPinecone() returns a near type match for the Pinecone client
  const vectorResponse = await pinecone.Index('main').query(vectorQuery);
  const snippets = // process vectorResponse
  
  const response = await openai.completions.create({
    model: 'text-davinci-003',
    temperature: 0.7,
    // We do modify OpenAI in one way, splitting prompt into template and inputs
    // this allows us to monitor metrics changes due to the template
    promptTemplate: `Context:\n\n{{ snippets }}\n\n{{ input}}`,
    promptInputs: {
        input,
        snippets,
    },
  });
  
  // Data is submitted asynchronously
  await runner.submit();
  
  return response;
}