Test runners - Create with local data

TypeScript only

This feature is currently only available in our TypeScript SDK. We have not implemented this functionality in Python yet.

The createTestRunners() function allows you to create test runners using local test data, rather than using test case data defined in datasets. This is useful when you want to run evaluations on custom, locally-defined test cases.

Use cases

• Running evaluations on custom test data not stored in Gentrace datasets
• Quick prototyping and testing of AI models with local inputs
• Integrating external test case sources into your Gentrace workflow

Example

typescript
import { createTestRunners, LocalTestData, Pipeline, submitTestRunners } from "@gentrace/core";
import { initPlugin } from "@gentrace/openai";
 
// Initialize Gentrace and OpenAI plugin
const plugin = await initPlugin({ apiKey: process.env.OPENAI_KEY });
 
const pipeline = new Pipeline({
  slug: "your-pipeline-slug",
  plugins: { openai: plugin }
});
 
// Define local test data
const localData: LocalTestData[] = [
  {
    name: "Test Case 1",
    inputs: { prompt: "Convert this sentence to JSON: John is 10 years old." },
  },
  // Add more test cases as needed
];
 
// Create test runners using local data
const testRunners = createTestRunners(pipeline, localData);
 
// Process test cases
for (const [runner, testCase] of testRunners) {
  // Use runner.openai to make API calls
  const completion = await runner.openai.chat.completions.create({
    model: "gpt-3.5-turbo",
    messages: [{ role: "user", content: testCase.inputs.prompt }],
  });
 
  // Optionally, add local evaluations
  runner.addEval({
    name: "example-eval",
    value: 0.8,
    // Add more evaluation details as needed
  });
}
 
// Submit test runners
const result = await submitTestRunners(pipeline, testRunners);
console.log("Test result ID:", result.resultId);

Arguments

pipeline: Pipeline

The pipeline instance to associate with the test runners.

localData: LocalTestData[]

An array of local test data objects. Each object should have the following structure:

typescript
interface LocalTestData {
  name: string;
  inputs: Record<string, any>;
}

Return value

Returns an array of PipelineRunLocalDataTuple objects, which are tuples containing a PipelineRun instance and the corresponding LocalTestData data.

Integration with submitTestRunners()

After creating test runners with createTestRunners(), you can use submitTestRunners() to submit the results to Gentrace. This allows you to run evaluations and store the results in Gentrace.

typescript
const result = await submitTestRunners(pipeline, testRunners);
console.log("Test result ID:", result.resultId);