PipelineRun class - addEval()

The PipelineRun class captures the output for a generative invocation. addEval() is a method under this class that allows you to add local evaluations to your pipeline run.

TypeScript Only

The addEval() function is only available in the TypeScript SDK.

Usage

The addEval() method is used to add local evaluations to your pipeline run. These evaluations can be used to assess the quality, performance, or other aspects of your generative AI outputs.

Example

typescript
// Penalizes emails that are too short or too long, with the ideal length around 100 words
const calculateScore = (count: number) => {
  const mean = 100;
  const stdDev = 30;
  const maxScore = 1;
  
  const zScore = Math.abs(count - mean) / stdDev;
  return Math.max(0, maxScore * Math.exp(-0.5 * Math.pow(zScore, 2)));
};
 
const wordCount = result.choices[0].message.content.split(/\s+/).length;
const lengthScore = calculateScore(wordCount);
 
// Adding a custom evaluation
runner.addEval({
  name: "response-length-score",
  value: lengthScore,
});

Arguments

The addEval() method takes a single argument of type LocalEvaluation:

typescript
interface LocalEvaluation {
  name: string; // Evaluation identifier
  value: number; // Evaluation score between 0 and 1
  label?: string | null; // Evaluation label to describe the evaluation (e.g. Positive/Negative or Inaccurate/Factually consistent)
  debug?: LocalEvaluationDebug; // Additional debug info

Return Value

The addEval() method does not return a value. It adds the evaluation to the internal state of the PipelineRun instance.

Local evaluations are submitted to Gentrace during test result submission (view the next section below).

Integration with submitTestRunners()

We generally recommend using addEval() with getTestRunners() and submitTestRunners(). These functions provide runners for each test case in your dataset. For more information on how to use these functions, see the Test runners - Get and Submit documentation.

Once you invoke addEval() for a runner, the local evaluations are automatically included in the submitted test results.

typescript
const response = await submitTestRunners(pipeline, pipelineRunTestCases, {
  triggerRemoteEvals: true, // This will include both remote and local evaluations
});