Manage test data with test cases

Test cases are examples that your generative AI pipeline might encounter. Test cases are uniquely associated with a pipeline. A test case contains:

• A unique name
• Inputs that will be passed to your AI pipeline
• Expected output (optional, depending on the evaluators that you need)
• Expected output steps (optional, also depending on the evaluators that you need)

Schema

This section breaks down the test case schema in more detail.

Name

Simple, human-readable name for your test case.

Inputs

Inputs are the parameters to your AI pipeline, expressed as a JSON string.

Let's say you have a simple AI pipeline (a function with an OpenAI invocation) that composes an email. The function accepts a sender, receiver, and query as string.

TypeScript

typescript
import { init } from "@gentrace/core"
import { OpenAI } from "@gentrace/openai";
 
init({
  apiKey: 'my-gentrace-api-key', // TODO: Add your Gentrace API key
})
 
const openai = new OpenAI({
  apiKey: 'my-open-ai-api-key', // TODO: Add your OpenAI API key
});
 
export const compose = async ({
  sender,
  receiver,
  query,
}: {
  sender: string;
  receiver: string;
  query: string;
}) => {
  const response = await openai.chat.completions.create({
    pipelineId: "draft",
    model: "gpt-3.5-turbo",
    temperature: 0,
    messages: [
      {
        role: "system",
        content: `Write a concise and complete email from ${sender} to ${receiver} ${query}.`,
      },
    ],
  });
  return {
    content: response.choices[0]!.message!.content,
    pipelineRunId: response.pipelineRunId!,
  };
};
 

Python

python
import openai
def compose(sender, receiver, query):
    prompt = f"Write a concise and complete email from {sender} to {receiver} {query}."
    chat_completion = openai.chat.completions.create(
      model="gpt-3.5-turbo",
      messages=[{"role": "user", "content": prompt}]
    )
    return chat_completion.choices[0].message.content

The below JSON object input string could represent the parameters to this pipeline.

json
{
  "query": "bragging about superiority",
  "sender": "[email protected]",
  "receiver": "[email protected]"
}

Exact match required

Each key from the inputs should exactly match what your AI pipeline expects. The inputs must be a JSON object. Arrays or primitive types (e.g. number, strings, booleans) are not permitted.

Expected outputs (optional)

This object captures the expected, ideal outputs of your pipeline.

Referring to the code example in the prior section, the expected output would be the ideal chat completion string returned from the function. Here's an example string that could work well as the expected output for the case.

Dear Joker, 
  
It has come to our attention that instances of bragging about superiority with respect to the 
Justice League have been made, and we want to emphasize that such behavior is not condoned 
or representative of our organization's values of justice, respect, and collaboration.
Best,
Superman

This would need to be inserted as a JSON structure, eg { "value": "Dear Joker..." }

Managing test cases

In the UI (small datasets)

If you only need to specify a few test cases, you can create them directly from the UI by selecting "New test case".

From CSV

You can also bulk import test cases from a CSV. Navigate to the test cases page and click the Import from CSV button.

Select the relevant CSV file.

Pay attention to the in-app instructions for dealing with headers.

With API/SDK

We expose a few methods to perform CRUD operations on test cases. This is helpful for creating internal workflows to manage test cases.

Refer to the following SDK methods:

• Create
• Bulk create
• Get
• Bulk get
• Update

And corresponding API endpoints:

• Create
• Update
• Get
• Bulk get

Adding images / other files for multi-modal

If you're evaluating multi-modal pipelines, you can upload images or other files to Gentrace and use them as inputs to your test cases.

Alternatively, you can link to them instead (see Option 2 below).

Option 1: Upload files

If your pipeline depends on file inputs (e.g. images, PDFs), you can upload your files to our object storage. We then return an authenticated URL to add to your test case inputs.

TypeScript

python
import os
import gentrace
from dotenv import load_dotenv
load_dotenv()
gentrace.init(
    api_key=os.getenv("GENTRACE_API_KEY"), host="http://localhost:3000/api/v1"
)
GENTRACE_PIPELINE_SLUG = "main"
with open("/home/user/gentrace-icon.png", "rb") as f:
    # This SDK method receives a file handle and returns an authenticated URL
    url = gentrace.upload_file(f)
    print("Gentrace file URL: ", url)
    gentrace.create_test_case(
      	# Pipeline slug
        GENTRACE_PIPELINE_SLUG,
        {
            "name": 'Gentrace Icon',
            'inputs': {
              	# Any Gentrace-uploaded URL will be rendered in a pretty format in the UI
                'iconUrl': url,
            },
            "expectedOutputs": {
                "value": "Gentrace logo"
            }
        }
    )

When viewing the test cases with Gentrace file URLs, we detect image extensions and render them directly in the UI.

Uploading file content bytes

If your file does not have a presence on the filesystem, you can directly upload the bytes to Gentrace. This approach requires you to name your file.

Python only

Uploading file content bytes is only supported in Python at this time.

Python

python
import gentrace
gentrace.init(
    api_key=os.getenv("GENTRACE_API_KEY")
)
with open("examples/files/gentrace-icon.png", "rb") as f:
    # Remember to specify the file extension! The Gentrace UI relies on the file
    # extension to render file contents correctly.
    url = gentrace.upload_bytes("gentrace-icon.png", f.read())
print("Uploaded URL: ", url)
gentrace.create_test_case(
    "main",
    {
        "name": 'Gentrace Icon',
        'inputs': {
            'imageUrl': url,
        },
        "expectedOutputs": {
            "value": "Gentrace logo"
        }
    }
)

Python

typescript
import { init, uploadBuffer, createTestCase } from "@gentrace/core";
import fs from "fs/promises";
 
init({
  apiKey: process.env.GENTRACE_API_KEY,
});
 
async function upload() {
  const buffer = await fs.readFile("./icon.png");
  // This SDK method receives a file name + buffer and returns an authenticated URL
  const url = await uploadBuffer("icon.png", buffer);
  
  const caseId = await createTestCase({
    pipelineSlug: "main",
    name: "Gentrace Icon",
    inputs: {
      // Any Gentrace-uploaded URL will be rendered in a pretty format in the UI
      iconUrl: url
    },
    expectedOutputs: {
      value: "Gentrace Icon"
    },
  });
 
  console.log("Case ID", caseId);
}
 
upload();
 

When viewing the test cases with Gentrace file URLs, we detect image extensions and render them directly in the UI.

Retrieving files

All uploaded files require authentication by an API key. When pulling test cases for a Gentrace pipeline, you need to construct an authenticated HTTP request to download the files associated with each case.

Here's a script that:

• Pulls test cases for a Gentrace pipeline
• Downloads the Gentrace-uploaded images from the input URL
• Runs the image data through our AI business logic
• Submits the outputs for grading

TypeScript

python
import os
from urllib.parse import urlparse
import gentrace
import requests
from dotenv import load_dotenv
# Import your AI pipeline
from ai.pipelines import image_to_words
GENTRACE_PIPELINE_SLUG = "main"
load_dotenv()
gentrace.init(
    api_key=os.getenv("GENTRACE_API_KEY")
)
cases = gentrace.get_test_cases(pipeline_slug=GENTRACE_PIPELINE_SLUG)
outputs = []
for case in cases:
    image_url = case.get("inputs").get("imageUrl")
    # Image URLs are authenticated. You must provide an API key as the bearer token.
    headers = {
        'Authorization': 'Bearer {}'.format(os.getenv("GENTRACE_API_KEY"))
    }
    response = requests.get(image_url, headers=headers)
    
    # Run the AI pipeline on the raw image content
    image_description = image_to_words(response.content)
    outputs.append({
      "value": image_description
    })
response = gentrace.submit_test_result(GENTRACE_PIPELINE_SLUG, cases, outputs)
print(response["resultId"])
  

Python

typescript
import { init, submitTestResult, getTestCases } from "@gentrace/core";
import { imageToWords } from "../api/pipelines";
 
const GENTRACE_PIPELINE_SLUG = "main";
const GENTRACE_API_KEY = process.env.GENTRACE_API_KEY;
 
init({
  apiKey: GENTRACE_API_KEY,
});
 
async function runTest() {
  const cases = await getTestCases(GENTRACE_PIPELINE_SLUG);
 
  const outputs: Record<string, any>[] = [];
 
  for (const testCase of cases) {
    const url = testCase.inputs.imageUrl;
    const response = await fetch(url, {
      method: "GET",
      headers: {
        Authorization: `Bearer ${GENTRACE_API_KEY}`,
      },
    });
 
    const blob = await response.blob();
    const words = await imageToWords(blob);
    outputs.push({
      value: words,
    });
  }
 
  const response = await submitTestResult(
    GENTRACE_PIPELINE_SLUG,
    cases,
    outputs,
  );
 
  console.log("Result ID:", response.resultId);
}
 
runTest();
 

Option 2: Link to files

In order to link to external images or files and have them render in Gentrace, you need to authorize external domains.

Administrators can do this by navigating to security settings, scrolling to "Authorized file URL domains," and pressing "Add domain."

Once you've added a domain, you can place URLs from that domain in your test case inputs. These URLs will render in the UI and be accessible as URLs in your pipeline.