Test results - Run test

TypeScript

The runTest() function also creates a test result but simplifies the operation by pulling test cases and submitting the test result in a single function.

Python

The run_test() function also creates a test result but simplifies the operation by pulling test cases and submitting the test result in a single function.

As part of this process, you specify a PipelineRun class instance that captures intermediate generative steps that will be associated with the test result.

Learn more about how to use this function in a guided way in the tutorial.

Arguments

typescript

pipelineSlug

python

pipeline_slug

The slug of the pipeline for which test results are being submitted. This string uniquely identifies the pipeline within the system.

typescript

testFunction

python

test_function

This function accepts a test case as a parameter. The return value of this function is an array where the first element is the output of the test case and the second element is the PipelineRun class instance that captures intermediate generative steps.

typescript

typescript
await runTest(PIPELINE_SLUG, 
  async (testCase) => {
    const runner = pipeline.start();
  
    const outputs = await runner.measure(
      (inputs) => {
        return {
          example: generateAiResponse(inputs),
        };
      },
      [testCase.inputs],
    );
  
    await runner.submit();
  
    // 🚧 Passing the runner back from this function is very important
    return [outputs, runner];
  }
);

python

Return value

This endpoint returns a simple object with the test result ID as a UUID string. Here's an example response structure.

json
{
  "resultId": "FACB6642-4725-4FAE-9323-634E72533C89"
}

You can then use this ID to retrieve the test result using the getTestResult() function or check the status with the getTestResultStatus() function.

Example

typescript

typescript
import { init, runTest, Pipeline } from "@gentrace/core";
import { generateAiResponse } from "../pipelines";
 
const PIPELINE_SLUG = "my-pipeline";
 
init({
  apiKey: process.env.GENTRACE_API_KEY,
});
 
const pipeline = new Pipeline({
  slug: PIPELINE_SLUG,
})
 
await runTest(
  PIPELINE_SLUG,
  async (testCase) => {
    const runner = pipeline.start();
 
    const outputs = await runner.measure(
      (inputs) => {
        return {
          example: generateAiResponse(inputs),
        };
      },
      [testCase.inputs],
    );
 
    await runner.submit();
 
    // 🚧 Passing the runner back from this function is very important
    return [outputs, runner];
  },
);

python

python
import os
import gentrace
from pipelines import generate_ai_response # TODO import your pipeline
PIPELINE_SLUG = "my-pipeline"
gentrace.init({
    "apiKey": os.environ["GENTRACE_API_KEY"],
})
pipeline = gentrace.Pipeline(PIPELINE_SLUG)
pipeline.setup()
def generate(test_case):
    runner = pipeline.start()
    
    output = runner.measure(
        lambda inputs: generate_ai_response(inputs),
        inputs=test_case.get("inputs")
    )
   
    # 🚧 Passing the runner back from this function is very important 
    return [output, runner]
result = gentrace.run_test(PIPELINE_SLUG, generate)
print("Result: ", result)