Experiments

Use experiments to quickly tweak parameters in your generative AI pipeline and then run your pipeline using these changed parameters.

🛑Alpha

Experiments is currently in alpha and may undergo significant changes.

No Python support

We have not implemented this SDK function in Python yet.

Environment setup

Gentrace needs a connection to your environment to run the experiments with modified parameters through your pipeline.

Local setup

If you are running your experiments against a local version of your pipeline, you can open a websocket to Gentrace from your local machine. In your testing script, using the Typescript SDK:

typescript
import { init, listen } from '@gentrace/core';
init({
  apiKey: process.env.GENTRACE_API_KEY ?? '',
});
listen();

Server setup

To run experiments against a production or staging version of your pipeline, or if you would rather use a webhook over a websocket, you can register a webhook environment with Gentrace.

Create the environment in Settings > Environments. You must give your environment a unique name and add the webhook url that will be listening for incoming traffic.

Then within your code, setup the webhook at the path you specified.

typescript
import { init, handleWebhook } from "@gentrace/core";
import express from "express";
 
init({
  apiKey: process.env.GENTRACE_API_KEY ?? "",
  basePath: "http://localhost:3000/api",
});
 
// Create Express app
const app = express();
 
app.post("/", async (req: express.Request, res: express.Response) => {
  // Get the body of the request as a JSON object
  const body = req.body;
  await handleWebhook(body, (responseBody) => {
    res.status(200).json(responseBody);
  });
});
 
// Start the server
const PORT = process.env.PORT || 443;
app.listen(PORT, () => {
  console.log(`Server listening on port ${PORT}`);
});

Securing your webhook

Our webhook requests are secured through content hashing with a signature. We calculate the SHA256 HMAC signature from the body and include that signature in the x-gentrace-signature.

To verify the webhook, calculate the signature from the raw request body (not the JSON parsed body) using the webhook secret available on your environment settings.

Once you have created the environment, you can copy the webhook secret from the environments settings page.

Securely store your secret and use it in your webhook verification code:

typescript
import express from "express";
import { createHmac } from "crypto";
 
// Create Express app
const app = express();
 
// Verify the signature of the request
app.use(
  express.json({
    verify: (req: any, res: express.Response, buf, next) => {
      req.rawBody = buf;
      const webhookSecret = process.env.GENTRACE_WEBHOOK_SECRET ?? "";
 
      // Get the signature from the header
      const signature = req.header("x-gentrace-signature");
 
      if (!signature) {
        throw new Error("No signature provided");
      }
 
      const calculatedSignature = `sha256=${createHmac("sha256", webhookSecret)
        .update(req.rawBody)
        .digest("hex")}`;
 
      // Verify the signature
      if (signature !== calculatedSignature) {
        throw new Error("Invalid signature");
      }
    },
  }),
);
 
// Add error handling middleware
app.use(
  (
    err: Error,
    req: express.Request,
    res: express.Response,
    next: express.NextFunction,
  ) => {
    res.status(401).send("Unauthorized");
  },
);

Register parameters and interactions

Before tweaking the parameters you use in your pipeline, you must first specify what those parameters are. Three types of parameters are supported: string, number, enum (a list of strings) or template (a Mustache template).

You then register the interactions with your pipeline that uses those parameters. Wrap all of your pipeline functionality that you want to experiment on in a call to defineInteraction.

Here is an example for a numeric parameter in a pipeline that randomly guesses the year for a query.

typescript
import { defineInteraction, numericParameter } from '@gentrace/core';
import { z } from 'zod';
 
const randomYearParameter = numericParameter({
  name: 'Random component of year',
  defaultValue: 3,
});
 
const guessTheYear = defineInteraction({
  name: 'Guess the year',
  fn: async ({ query }) => {
    // Call you pipeline code here
    return Math.floor(Math.random() * randomYearParameter.getValue()) + 2022;
  },
  parameters: [randomYearParameter],
  // Optional - validates input types
  inputType: z.object({
    query: z.string(),
  }),
});

Here is an example for a template parameter in a pipeline that writes an email based on provided instructions.

typescript
import { defineInteraction, templateParameter } from '@gentrace/core';
import OpenAI from 'openai';
import { z } from 'zod';
 
const openai = new OpenAI({
  apiKey: process.env.OPENAI_KEY ?? "",
});
 
const writeEmailPromptParameter = templateParameter({
  name: 'Write email prompt',
  defaultValue:
    'Write an email to {{toEmail}} from ({{fromName}}) {{fromEmail}} according to these instructions: {{instructions}}',
});
 
const writeEmail = defineInteraction({
  name: 'Write email',
  fn: async ({ fromName, fromEmail, toEmail, instructions }) => {
    // Call your pipeline code here
    const completion = await openai.chat.completions.create({
      model: 'gpt-4o-mini',
      messages: [
        {
          role: 'user',
          content: writeEmailPromptParameter.render({
            fromName,
            fromEmail,
            toEmail,
            instructions,
          }),
        },
      ],
    });
    return {
      body: completion.choices[0].message.content,
    };
  },
  parameters: [writeEmailPromptParameter],
  // Optional - validates input types
  inputType: z.object({
    fromName: z.string(),
    fromEmail: z.string().email(),
    toEmail: z.string().email(),
    instructions: z.string(),
  }),
});

Here is an example for an enum parameter that you could use to test different models for your pipeline.

typescript
import { defineInteraction, enumParameter } from '@gentrace/core';
import { z } from 'zod';
 
const modelParameter = enumParameter({
  name: "AI Model",
  defaultValue: "GPT-4o",
  options: ["GPT-4o", "GPT-4o-mini", "claude-3.5-sonnet", "gemini-1.5-pro-002"],
});
 
const chooseModel = defineInteraction({
  name: "Choose model",
  fn: async ({ query }) => {
    // Call your pipeline code here, using the model provided
    return `I will use the model ${modelParameter.getValue()}.`;
  },
  inputType: z.object({
    query: z.string(),
  }),
  parameters: [modelParameter],
});
 

Interactions without parameters

You can also define interactions without providing any parameters if you would like to test local code variations, or experiment with different datasets.

typescript
import { defineInteraction } from '@gentrace/core';
 
const testInteraction = defineInteraction({
  name: 'YOUR_FEATURE',
  fn: async ({ blah }) => {
    // Insert calling your pipeline code here
    return {
      response: 'YOUR_RESPONSE',
    };
  },
});

Run experiments

Once you have your environment set up and your parameters and interactions registered, you can now run your experiments. To create a new experiment, navigate to your Pipeline > Test Results. Then click on the "New experiment" button in the top right corner.

You can choose which environment, dataset and interaction you would like to test. Then you can modify the parameters to override the defaults as you see fit.

Input validation

If you have provided input validation with your interaction, you will see validation fail for datasets that do not match your expected input type.

New result

Once you run your experiment you will see a new result in your result grid view.

If you view the metadata for this new experiment result, you can also start a new experiment based off of the parameters used in that result directly from the sidebar.