> ## Documentation Index
> Fetch the complete documentation index at: https://next.gentrace.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Unit tests

> Write unit tests for agents with `eval()` and `evalOnce()`

The Python `eval()` and TypeScript `evalOnce()` functions create individual test cases within an [`experiment()`](./experiments). These functions capture the execution of specific test logic, automatically creating [OpenTelemetry spans](https://opentelemetry.io/docs/concepts/signals/traces/#spans) with detailed tracing information, and associating results with the parent experiment.

These functions must be called within the context of an [`experiment()`](./experiments) and automatically create individual test spans for each evaluation.

<Tip>
  The Gentrace SDK automatically configures OpenTelemetry when you
  call `init()`. If you have an existing OpenTelemetry setup or need
  custom configuration, see the [manual setup
  guide](../reference/opentelemetry-guide).
</Tip>

## Basic usage

<CodeGroup dropdown>
  ```typescript theme={null}
  import { init, experiment, evalOnce, interaction } from 'gentrace';

  init({
    apiKey: process.env.GENTRACE_API_KEY,
  });

  // Define your AI function with interaction wrapper
  const myAIFunction = interaction(
    'my-ai-function',
    async (input: string): Promise<string> => {
      // Your AI logic here - e.g., call to OpenAI, Anthropic, etc.
      return 'This is a sample AI response';
    },
  );

  // Basic evalOnce usage within an experiment
  experiment(async () => {
    await evalOnce('simple-accuracy-test', async () => {
      const input = 'What is 2 + 2?';
      const result = await myAIFunction(input);
      const expected = '4';

      return {
        input,
        result,
        expected,
        passed: result.includes(expected),
      };
    });
  });
  ```

  ```python theme={null}
  import asyncio
  import os
  from gentrace import init, experiment, eval, interaction

  init(api_key=os.environ["GENTRACE_API_KEY"])

  # Define your AI function with interaction wrapper
  @interaction(name="my-ai-function")
  async def my_ai_function(input_text: str) -> str:
      # Your AI logic here - e.g., call to OpenAI, Anthropic, etc.
      return "This is a sample AI response"

  @experiment
  async def my_experiment() -> None:
      @eval(name="simple-accuracy-test")
      async def simple_accuracy_test() -> dict:
          input_text = "What is 2 + 2?"
          result = await my_ai_function(input_text)
          expected = "4"

          return {
              "input": input_text,
              "result": result,
              "expected": expected,
              "passed": expected in result
          }

      await simple_accuracy_test()

  # Run the experiment
  asyncio.run(my_experiment())
  ```
</CodeGroup>

## Overview

Individual evaluations in Gentrace represent specific test cases or validation steps within your AI pipeline testing. The `eval()` and `evalOnce()` functions provide comprehensive test execution capabilities:

### Key features

<ResponseField name="OpenTelemetry integration" type="feature">
  Generate [OpenTelemetry
  spans](https://opentelemetry.io/docs/concepts/signals/traces/) with
  detailed execution tracing for each test case
</ResponseField>

<ResponseField name="Automatic data capture" type="feature">
  Record the return value of `eval()` or `@eval()`-decorated functions
  as the output for each test span
</ResponseField>

<ResponseField name="Experiment association" type="feature">
  Link test cases to the parent experiment context for proper grouping
  and analysis
</ResponseField>

<ResponseField name="Error resilience" type="feature">
  Preserve full error information in traces while allowing tests to
  continue running
</ResponseField>

<ResponseField name="Execution flexibility" type="feature">
  Support both synchronous and asynchronous test functions seamlessly
</ResponseField>

## Parameters

<CodeGroup dropdown>
  ```typescript theme={null}
  /**
   * Run a single evaluation test case
   *
   * @param spanName - A descriptive name for the test case,
   *                   used for tracing and reporting
   * @param callback - The function containing your test logic.
   *                   Can be synchronous or asynchronous
   *
   * @returns A Promise<TResult | null> that resolves with:
   *          - The result of your callback function on success
   *          - null if an error occurs (errors are captured
   *            in the OpenTelemetry span)
   */
  function evalOnce<TResult>(
    spanName: string,
    callback: () => TResult | null | Promise<TResult | null>,
  ): Promise<TResult | null>;
  ```

  ```python theme={null}
  def eval(
      *,
      name: str,
      metadata: Optional[Dict[str, Any]] = None,
  ) -> Callable[[Callable[P, Any]], Callable[P, Coroutine[Any, Any, Any]]]:
      """
      Decorator for evaluating a function as a test case

      Args:
          name: A descriptive name for the test case,
                used for tracing and reporting
          metadata: Custom metadata to associate with
                    this evaluation's span

      Returns:
          A decorator that wraps your function.
          The wrapped function, when called, will:
          - Execute within an OpenTelemetry span
          - Capture inputs, outputs, and any errors
          - Return the original function's result
      """

  ```
</CodeGroup>

### Multiple test cases with different evaluation types

<CodeGroup dropdown>
  ```typescript theme={null}
  import { callAIModel } from './models';
  import { experiment, evalOnce, interaction } from 'gentrace';

  const myAIFunction = interaction(
    'multi-test-function',
    async (input: string): Promise<string> => {
      return await callAIModel(input);
    },
  );

  function calculateAccuracy(result: string, expected: string): number {
    const resultWords = result.toLowerCase().split(/\s+/);
    const expectedWords = expected.toLowerCase().split(/\s+/);
    const matches = expectedWords.filter((word) =>
      resultWords.includes(word),
    );
    return matches.length / expectedWords.length;
  }

  experiment(async () => {
    // Test accuracy
    await evalOnce('accuracy-test', async () => {
      const input = 'What is machine learning?';
      const expected = 'machine learning is artificial intelligence';
      const result = await myAIFunction(input);
      const accuracy = calculateAccuracy(result, expected);

      return {
        input,
        result,
        expected,
        accuracy,
        passed: accuracy >= 0.7,
      };
    });

    // Test latency
    await evalOnce('latency-test', async () => {
      const start = Date.now();
      const result = await myAIFunction('Quick test input');
      const latency = Date.now() - start;

      return {
        result,
        latency,
        threshold: 2000,
        passed: latency < 2000,
      };
    });
  });
  ```

  ```python theme={null}
  import time
  import asyncio
  from models import call_ai_model

  @interaction(name="multi-test-function")
  async def my_ai_function(input_text: str) -> str:
      return await call_ai_model(input_text)

  def calculate_accuracy(result: str, expected: str) -> float:
      result_words = set(result.lower().split())
      expected_words = set(expected.lower().split())
      matches = result_words.intersection(expected_words)
      return len(matches) / len(expected_words) if expected_words else 0.0

  @experiment
  async def comprehensive_experiment() -> None:
      @eval(name="accuracy-test")
      async def accuracy_test() -> dict:
          input_text = "What is machine learning?"
          expected = "machine learning is artificial intelligence"
          result = await my_ai_function(input_text)
          accuracy = calculate_accuracy(result, expected)

          return {
              "input": input_text,
              "result": result,
              "expected": expected,
              "accuracy": accuracy,
              "passed": accuracy >= 0.7
          }

      @eval(name="latency-test")
      async def latency_test() -> dict:
          start = time.time()
          result = await my_ai_function("Quick test input")
          latency = time.time() - start

          return {
              "result": result,
              "latency": latency,
              "threshold": 2.0,
              "passed": latency < 2.0
          }

      await accuracy_test()
      await latency_test()
  ```
</CodeGroup>

## OTEL span error integration

When errors occur during individual evaluations:

* **Automatic Capture**: All validation errors and interaction exceptions are automatically captured as [span events](https://opentelemetry.io/docs/concepts/signals/traces/#span-events)
* **Individual Spans**: Each evaluation gets its own span, so errors are isolated to specific test cases
* **Continued Processing**: Failed evaluations don't stop the execution of other evaluations in the experiment
* **Error Attributes**: Error messages, types, and metadata are recorded as [span attributes](https://opentelemetry.io/docs/concepts/signals/traces/#attributes)
* **Span Status**: Individual evaluation spans are marked with [`ERROR` status](https://opentelemetry.io/docs/concepts/signals/traces/#span-status) when exceptions occur
* **Error Handling**: See [OpenTelemetry error recording](https://opentelemetry.io/docs/specs/semconv/general/recording-errors/) and [exception recording](https://opentelemetry.io/docs/specs/otel/trace/exceptions/) for more details

## Requirements

* **Gentrace SDK Initialization**: Must call [`init()`](../getting-started/initialization) with a valid API key. The SDK automatically configures OpenTelemetry for you. For custom OpenTelemetry setups, see the [manual setup guide](../reference/opentelemetry-guide)
* **Experiment Context**: Must be called within an [`experiment()`](./experiments) function
* **Valid Pipeline ID**: When using an explicit pipeline ID, it must be valid

## Context requirements

Both `eval()` and `evalOnce()` **must be called within an active experiment context**. They automatically:

1. **Retrieve experiment context** from the parent `experiment()` function
2. **Associate spans** with the experiment ID for proper grouping
3. **Inherit experiment metadata** and configuration

<CodeGroup dropdown>
  ```typescript theme={null}
  // ❌ This will throw an error - no experiment context
  await evalOnce('invalid-test', async () => {
    return 'This will fail';
  });

  // ✅ Correct usage within experiment context
  experiment(async () => {
    await evalOnce('valid-test', async () => {
      return { message: 'This will work', passed: true };
    });
  });
  ```

  ```python theme={null}
  # ❌ This will throw an error - no experiment context
  @eval(name="invalid-test")
  async def invalid_test() -> str:
      return "This will fail"

  await invalid_test()  # RuntimeError: must be called within @experiment context

  # ✅ Correct usage within experiment context
  @experiment
  async def valid_experiment() -> None:
      @eval(name="valid-test")
      async def valid_test() -> dict:
          return {"message": "This will work", "passed": True}

      await valid_test()
  ```
</CodeGroup>

## OpenTelemetry integration

The evaluation functions create rich [OpenTelemetry spans](https://opentelemetry.io/docs/concepts/signals/traces/#spans) with comprehensive tracing information:

### Span attributes

<ResponseField name="gentrace.experiment_id" type="string" required>
  Links the evaluation to its parent experiment
</ResponseField>

<ResponseField name="gentrace.test_case_name" type="string" required>
  The name provided to the evaluation function
</ResponseField>

<ResponseField name="Error information" type="automatic">
  Automatic error type and message capture
</ResponseField>

### Span events

<ResponseField name="gentrace.fn.output" type="event">
  Records function outputs for result tracking
</ResponseField>

<ResponseField name="Exception events" type="event">
  Automatic exception recording with full stack traces
</ResponseField>

### Example span structure

```mermaid theme={null}
graph TD
    EXP["Experiment Span<br/>experiment-name"]

    EVAL1["Evaluation Span<br/>accuracy-test"]
    EVAL2["Evaluation Span<br/>latency-test"]

    ATTR1["Attributes:<br/>gentrace.experiment_id: exp-123<br/>gentrace.test_case_name: accuracy-test<br/>custom.metadata: value"]

    EVENTS1["Events:<br/>gentrace.fn.args: input data<br/>gentrace.fn.output: accuracy 0.95"]

    CHILD1["Child Spans<br/>from interaction calls"]

    EXP --> EVAL1
    EXP --> EVAL2

    EVAL1 --> ATTR1
    EVAL1 --> EVENTS1
    EVAL1 --> CHILD1

    EVAL2 --> |...|EVAL2_DETAILS[Similar structure]

    style EXP fill:#2E1F25,stroke:#53242E,color:#DDDDDD
    style EVAL1 fill:#1A2535,stroke:#18375D,color:#DDDDDD
    style EVAL2 fill:#1A2535,stroke:#18375D,color:#DDDDDD
    style ATTR1 fill:#1D2E33,stroke:#205258,color:#DDDDDD
    style EVENTS1 fill:#312724,stroke:#5E3D2B,color:#DDDDDD
    style CHILD1 fill:#2B3222,stroke:#4B5D25,color:#DDDDDD
    style EVAL2_DETAILS fill:#212327,stroke:#3D4044,color:#DDDDDD
```

## Related functions

* [`init()`](../getting-started/initialization) - Initialize the Gentrace SDK
* [`interaction()`](../tracing/interactions) - Instrument AI functions for tracing within experiments
* [`evalDataset()`](./dataset-tests) / `eval_dataset()` - Run tests against a dataset within an experiment
* [`experiment()`](./experiments) - Create experiment contexts for grouping evaluations
* [`traced()`](../tracing/traced) - Alternative approach for tracing functions