Skip to main content
Version: 4.5.0

Monitoring (OpenAI) quickstart

This guide covers how to get started with Gentrace for monitoring OpenAI. Gentrace monitors your logged OpenAI requests and tracks speed, cost, and aggregate statistics.

Installation

🛑Server-side only

Please only use this library on the server-side. Using it on the client-side will reveal your API key.

First, install our core package.

bash
# Execute only one, depending on your package manager
npm i @gentrace/core
yarn add @gentrace/core
pnpm i @gentrace/core

If you want to use our provider SDK handlers, you must install our associated plugin SDKs. These SDKs have a direct dependency on the officially supported SDK for their respective providers. We type match the official SDK whenever possible.

Requires @gentrace/openai@v4 or @gentrace/openai@v3

This section requires Gentrace's official OpenAI plugin. The plugin version matches the major version of the official OpenAI Node.JS SDK.

shell
# For OpenAI v4 (the new version)
npm install @gentrace/openai@v4
# For OpenAI v3 (the older version)
npm install @gentrace/openai@v3

These NPM packages will only work with Node.JS versions >= 16.16.0.

Usage

We designed our SDKs to mostly preserve the original interface to OpenAI's client library. You can simply insert the following lines of code before your OpenAI invocations.

typescript
import { init } from "@gentrace/core";
import { OpenAI } from "@gentrace/openai";
 
// This function globally initializes Gentrace with the appropriate
// credentials. Constructors like OpenAI() will transparently use
// these credentials to authenticate with Gentrace.
init({
apiKey: process.env.GENTRACE_API_KEY
});
 
const openai = new OpenAI({
apiKey: process.env.OPENAI_KEY,
});

The OpenAI class is virtually identical to the equivalents in the official SDK.

You can then execute your OpenAI functions against the openai handle directly.

typescript
async function createEmbedding() {
const embeddingResponse = await openai.embeddings.create({
model: "text-embedding-ada-002",
input: "Example input",
// IMPORTANT: Supply a Gentrace Pipeline slug to track this invocation
pipelineSlug: "create-test-embedding"
});
console.log("Pipeline run ID: ", embeddingResponse.pipelineRunId);
}
createEmbedding();
caution

You should provide a Pipeline slug as a request parameter to any method that you want to instrument. This ID associates OpenAI invocations to that identifier on our service. If you omit the slug, we will not track telemetry for that invocation.

The PipelineRun ID provided by the OpenAI create() return value is from the Gentrace SDK. Our SDK provides this for you to uniquely associate feedback with AI generated content. If you do not provide a Pipeline slug, the create() functions will not return a PipelineRun ID.

Prompt templates for the openai.completions.create() interface

One difference between the Gentrace-instrumented SDK and the official SDK is how prompts are specified for openai.completions.create() requests.

In the official version of the SDK, you must specify the prompt as a string. In our SDK, you must specify a prompt template and it's associated inputs. We use Mustache templating with the Mustache.js library to render the prompt templates.

typescript
// ❌ Official SDK invocation
await openai.completion.create({
...DEFAULT_PARAMS,
prompt: "What's the trendiest neighborhood in New York City?"
});
// ✅ Gentrace OpenAI Handle
await openai.completions.create({
...DEFAULT_PARAMS,
promptTemplate: "What's {{ interestingFact }} in {{ location }}?",
promptInputs: {
interestingFact: "the trendiest neighborhood",
location: "New York City"
},
});

Note: We still allow you to specify the original prompt parameter if you want to incrementally migrate your invocations.

Consult OpenAI's Node.JS SDK documentation for more details to learn more about the original SDK.

Content templates for the openai.chat.completions.create() interface

The other difference between the Gentrace-instrumented SDK and the official SDK is how prompts are specified for openAi.chat.completion.create() requests.

In the official version of the SDK, you specify your chat completion input as an object array with role and content key-pairs defined.

typescript
// ❌ Official OpenAI SDK invocation
const chatCompletionResponse = await openai.chat.completions.create({
messages: [
{
role: "user",
content: "Hello Vivek!"
},
],
model: "gpt-3.5-turbo"
});

In our SDK, if part of the content is dynamically generated, you should instead create contentTemplate and contentInputs key-pairs to separate the static and dynamic information, respectively. This is helpful to better display the generation in our UI and internally track version changes.

We use Mustache templating with the Mustache.js library to render the final content that is sent to OpenAI.

typescript
// ✅ Gentrace-instrumented OpenAI SDK
const chatCompletionResponse = await openai.chat.completions.create({
messages: [
{
role: "user",
contentTemplate: "Hello {{ name }}!",
contentInputs: { name: "Vivek" },
},
],
model: "gpt-3.5-turbo",
pipelineSlug: "testing-pipeline-id",
});

Note: We still allow you to specify the original content key-value pair in the dictionary if you want to incrementally migrate your invocations.

Consult OpenAI's Node.JS SDK documentation for more details to learn more about the original SDK.

Streaming

We transparently wrap OpenAI's Node streaming functionality.

typescript
// Imports and initialization truncated
async function main() {
const streamChat = await openai.chat.completions.create({
model: 'gpt-4',
messages: [{ role: 'user', content: 'Say this is a test' }],
stream: true,
});
// This PipelineRun ID actually hasn't been created on the server yet.
// It's created asynchronously after the final stream event is processed.
console.log("Pipeline run ID: ", streamChat.pipelineRunId);
for await (const part of streamChat) {
console.log(part.choices[0]?.delta?.content || '');
}
// Stream data is coalesced and sent to our server.
}
main();
caution

We measure the total time a generation takes from the first byte received from iterating on the stream to the last event yielded from the stream.

Before sending the streamed events to Gentrace, we coalesce the streamed payload as a single payload to improve readability.

Continue