> ## Documentation Index
> Fetch the complete documentation index at: https://next.gentrace.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Dataset tests

> Run evaluations against a dataset using `evalDataset()` and `eval_dataset()`

The Python `eval_dataset()` and TypeScript `evalDataset()` functions run a series of evaluations against a dataset using a provided [`interaction()`](../tracing/interactions) function.

These functions must be called within the context of an [`experiment()`](./experiments) and automatically create individual test spans for each test case in the dataset.

<Tip>
  The Gentrace SDK automatically configures OpenTelemetry when you
  call `init()`. If you have an existing OpenTelemetry setup or need
  custom configuration, see the [manual setup
  guide](../reference/opentelemetry-guide).
</Tip>

## Basic usage

<CodeGroup dropdown>
  ```typescript theme={null}
  import { init, experiment, evalDataset, testCases, interaction } from 'gentrace';
  import { OpenAI } from 'openai';

  init({
    apiKey: process.env.GENTRACE_API_KEY,
  });

  const DATASET_ID = process.env.GENTRACE_DATASET_ID!;

  const openai = new OpenAI({
    apiKey: process.env.OPENAI_API_KEY,
  });

  // Define your AI function
  async function processAIRequest(inputs: Record<string, any>) {
    // Your AI implementation here (e.g., call to OpenAI, Anthropic, custom model, etc.)

    return {
      result: "This is a sample AI response",
      metadata: { processed: true }
    };
  }

  // Wrap with interaction tracing (see interaction() docs)
  const tracedProcessAIRequest = interaction('Process AI Request', processAIRequest);

  // Basic dataset evaluation
  experiment(async () => {
    await evalDataset({
      data: async () => {
        const testCasesList = await testCases.list({ datasetId: DATASET_ID });
        return testCasesList.data;
      },
      interaction: tracedProcessAIRequest,
    });
  });
  ```

  ```python theme={null}
  import os
  import asyncio
  from gentrace import init, experiment, eval_dataset, test_cases_async, interaction, TestInput
  from openai import AsyncOpenAI

  init(api_key=os.environ["GENTRACE_API_KEY"])

  DATASET_ID = os.environ["GENTRACE_DATASET_ID"]

  openai = AsyncOpenAI(
      api_key=os.environ["OPENAI_API_KEY"]
  )

  # Define your AI function
  async def process_ai_request(inputs):
      # Your AI implementation here (e.g., call to OpenAI, Anthropic, custom model, etc.)

      return {
          "result": "This is a sample AI response",
          "metadata": {"processed": True}
      }

  # Wrap with interaction tracing (see interaction() docs)
  @interaction(name="Process AI Request")
  async def traced_process_ai_request(inputs):
      return await process_ai_request(inputs)

  @experiment
  async def dataset_evaluation() -> None:
      async def fetch_test_cases():
          test_case_list = await test_cases_async.list(dataset_id=DATASET_ID)
          return test_case_list.data

      await eval_dataset(
          data=fetch_test_cases,
          interaction=traced_process_ai_request,
      )

  # Run the experiment
  asyncio.run(dataset_evaluation())
  ```
</CodeGroup>

## Overview

Dataset evaluation functions allow you to:

1. **Run batch evaluations** against multiple test cases from a dataset
2. **Validate inputs** using optional schema validation ([Pydantic](https://docs.pydantic.dev/) for Python, [Zod](https://zod.dev/) or any [Standard Schema](https://github.com/standard-schema/standard-schema)-compliant schema library for TypeScript)
3. **Trace each test case** as individual [OpenTelemetry spans](https://opentelemetry.io/docs/concepts/signals/traces/#spans) within the experiment context
4. **Handle errors gracefully** with automatic span error recording
5. **Process results** from both synchronous and asynchronous data providers and interaction functions

## Parameters

### Function signature

<CodeGroup dropdown>
  ```typescript theme={null}
  function evalDataset<
    TSchema extends ParseableSchema<any> | undefined = undefined,
    TInput = TSchema extends ParseableSchema<infer TOutput>
      ? TOutput
      : Record<string, any>,
  >(options: EvalDatasetOptions<TSchema>): Promise<void>;
  ```

  ```python theme={null}
  async def eval_dataset(
      *,
      data: DataProviderType[InputPayload],
      schema: Optional[Type[SchemaPydanticModel]] = None,
      interaction: Callable[[Any], Union[TResult, Awaitable[TResult]]],
  ) -> Sequence[Optional[TResult]]
  ```
</CodeGroup>

### Parameters

<ResponseField name="options" type="EvalDatasetOptions" required>
  Configuration object containing data provider, interaction function, and optional schema

  <Expandable title="EvalDatasetOptions" defaultOpen>
    <ResponseField name="data" type="function" required>
      Function that returns test cases, either synchronously or asynchronously
    </ResponseField>

    <ResponseField name="schema" type="ParseableSchema">
      Schema object with a `parse()` method for input validation
    </ResponseField>

    <ResponseField name="interaction" type="function" required>
      The function to test against each test case. Typically created with [`interaction()`](../tracing/interactions).
    </ResponseField>
  </Expandable>
</ResponseField>

### TestInput Type

<CodeGroup dropdown>
  ```typescript theme={null}
  type TestInput<TInput extends Record<string, any>> = {
    name?: string | undefined;
    id?: string | undefined;
    inputs: TInput;
  };
  ```

  ```python theme={null}
  class TestInput(TypedDict, Generic[InputPayload], total=False):
      name: str
      inputs: InputPayload
  ```
</CodeGroup>

### Return Value

Returns a sequence of results from the interaction function. Failed test cases (due to validation errors) will have `None` values in the corresponding positions.

## Advanced usage

### With schema validation

Schema validation ensures that your test cases have the correct structure and data types before being passed to your interaction function.

Use [Zod](https://zod.dev/) for TypeScript or [Pydantic](https://docs.pydantic.dev/) for Python to define your input schemas.

<CodeGroup dropdown>
  ```typescript theme={null}
  import { z } from 'zod';
  import { evalDataset, testCases } from 'gentrace';
  import { OpenAI } from 'openai';

  const openai = new OpenAI({
    apiKey: process.env.OPENAI_API_KEY,
  });

  // Define input schema
  const InputSchema = z.object({
    prompt: z.string(),
    temperature: z.number().optional().default(0.7),
    maxTokens: z.number().optional().default(100),
  });

  experiment(async () => {
    await evalDataset({
      data: async () => {
        const testCasesList = await testCases.list({
          datasetId: DATASET_ID,
        });
        return testCasesList.data;
      },
      schema: InputSchema,
      interaction: async ({ prompt, temperature, maxTokens }) => {
        // Inputs are now typed and validated
        const response = await openai.chat.completions.create({
          model: 'gpt-4o',
          messages: [{ role: 'user', content: prompt }],
          temperature,
          max_tokens: maxTokens,
        });
        return response.choices[0].message.content;
      },
    });
  });
  ```

  ```python theme={null}
  import os
  from openai import AsyncOpenAI
  from pydantic import BaseModel
  from gentrace import eval_dataset, test_cases_async, TestInput

  openai = AsyncOpenAI(
      api_key=os.environ["OPENAI_API_KEY"]
  )

  class QueryInputs(BaseModel):
      prompt: str
      temperature: float = 0.7
      max_tokens: int = 100

  @experiment
  async def schema_validation_example() -> None:
      async def fetch_test_cases():
          test_case_list = await test_cases_async.list(dataset_id=DATASET_ID)
          return test_case_list.data

      async def ai_interaction(inputs):
          # Inputs are validated against QueryInputs schema
          response = await openai.chat.completions.create(
              model="gpt-4o",
              messages=[{"role": "user", "content": inputs["prompt"]}],
              temperature=inputs.get("temperature", 0.7),
              max_tokens=inputs.get("max_tokens", 100),
          )
          return response.choices[0].message.content

      results = await eval_dataset(
          data=fetch_test_cases,
          schema=QueryInputs,
          interaction=ai_interaction,
      )

      print(f"Processed {len(results)} test cases")
  ```
</CodeGroup>

### Custom data providers

You can provide test cases from any source by implementing a custom data provider function. Each data point must conform to the `TestInputs` structure from above.

This is useful when you want to use test cases from the Gentrace API via the test cases SDK or directly defining them in-line.

<CodeGroup dropdown>
  ```typescript theme={null}
  experiment(async () => {
    await evalDataset({
      data: () => [
        {
          name: 'Basic greeting test',
          id: 'test-1',
          inputs: { prompt: 'Say hello', temperature: 0.5 },
        },
        {
          name: 'Complex reasoning test',
          id: 'test-2',
          inputs: {
            prompt: 'Explain quantum computing',
            temperature: 0.8,
          },
        },
      ],
      interaction: async (inputs) => {
        return await myAIFunction(inputs);
      },
    });
  });
  ```

  ```python theme={null}
  @experiment
  async def custom_data_example() -> None:
      async def ai_interaction(inputs):
          return await my_ai_function(inputs)

      results = await eval_dataset(
          data=lambda: [
              TestInput(
                  name="Basic greeting test",
                  inputs={"prompt": "Say hello", "temperature": 0.5}
              ),
              TestInput(
                  name="Complex reasoning test",
                  inputs={"prompt": "Explain quantum computing", "temperature": 0.8}
              )
          ],
          interaction=ai_interaction,
      )
  ```
</CodeGroup>

## Error handling

The dataset evaluation functions handle errors gracefully and **automatically associate all errors and exceptions with [OpenTelemetry spans](https://opentelemetry.io/docs/concepts/signals/traces/#spans)**. When validation or interaction errors occur, they are captured as [span events](https://opentelemetry.io/docs/concepts/signals/traces/#span-events) and [attributes](https://opentelemetry.io/docs/concepts/signals/traces/#attributes).

<CodeGroup dropdown>
  ```typescript theme={null}
  experiment(async () => {
    await evalDataset({
      data: () => [
        { inputs: { prompt: 'Valid input' } },
        { inputs: { invalid: 'data' } }, // This might fail validation
        { inputs: { prompt: 'Another valid input' } },
      ],
      schema: z.object({ prompt: z.string() }),
      interaction: async (inputs) => {
        // This function might throw errors
        if (inputs.prompt.includes('error')) {
          throw new Error('Simulated error');
        }
        return await myAIFunction(inputs);
      },
    });

    // evalDataset continues processing other test cases even if some fail
    // All errors are automatically captured in OpenTelemetry spans
  });
  ```

  ```python theme={null}
  @experiment
  async def error_handling_example() -> None:
      def get_test_cases_with_errors():
          return [
              TestInput(inputs={"prompt": "Valid input"}),
              TestInput(inputs={"invalid": "data"}),  # This might fail validation
              TestInput(inputs={"prompt": "Another valid input"})
          ]

      async def ai_interaction_with_errors(inputs):
          # This function might raise exceptions
          if "error" in inputs.get("prompt", ""):
              raise ValueError("Simulated error")
          return await my_ai_function(inputs)

      results = await eval_dataset(
          data=get_test_cases_with_errors,
          schema=QueryInputs,
          interaction=ai_interaction_with_errors,
      )

      # results will contain None for failed test cases
      # All exceptions are automatically captured in OpenTelemetry spans
      successful_results = [r for r in results if r is not None]
      print(f"Successfully processed {len(successful_results)} out of {len(results)} test cases")
  ```
</CodeGroup>

### OTEL span error integration

When errors occur during dataset evaluation:

* **Automatic Capture**: All validation errors and interaction exceptions are automatically captured as [span events](https://opentelemetry.io/docs/concepts/signals/traces/#span-events)
* **Individual Spans**: Each test case gets its own span, so errors are isolated to specific test cases
* **Continued Processing**: Failed test cases don't stop the evaluation of other test cases
* **Error Attributes**: Error messages, types, and metadata are recorded as [span attributes](https://opentelemetry.io/docs/concepts/signals/traces/#attributes)
* **Span Status**: Individual test case spans are marked with [`ERROR` status](https://opentelemetry.io/docs/concepts/signals/traces/#span-status) when exceptions occur
* **Error Handling**: See [OpenTelemetry error recording](https://opentelemetry.io/docs/specs/semconv/general/recording-errors/) and [exception recording](https://opentelemetry.io/docs/specs/otel/trace/exceptions/) for more details

### Example span structure

```mermaid theme={null}
graph TD
    EXP["Experiment Span<br/>experiment-name"]

    DS["Dataset Evaluation<br/>evalDataset"]

    TC1["Test Case Span<br/>test-case-1-name"]
    TC2["Test Case Span<br/>test-case-2-name"]

    INT1["Interaction Span<br/>Process AI Request"]
    INT2["Interaction Span<br/>Process AI Request"]

    ATTR1["Attributes:<br/>gentrace.experiment_id: exp-123<br/>gentrace.test_case_name: test-case-1<br/>gentrace.test_case_id: tc-1"]

    EVENTS1["Events:<br/>gentrace.fn.args: test case inputs<br/>gentrace.fn.output: interaction result"]

    EXP --> DS
    DS --> TC1
    DS --> TC2

    TC1 --> INT1
    TC2 --> INT2

    TC1 --> ATTR1
    TC1 --> EVENTS1

    TC2 --> |...|TC2_DETAILS[Similar structure]

    style EXP fill:#2E1F25,stroke:#53242E,color:#DDDDDD
    style DS fill:#2B3222,stroke:#4B5D25,color:#DDDDDD
    style TC1 fill:#1A2535,stroke:#18375D,color:#DDDDDD
    style TC2 fill:#1A2535,stroke:#18375D,color:#DDDDDD
    style INT1 fill:#312724,stroke:#5E3D2B,color:#DDDDDD
    style INT2 fill:#312724,stroke:#5E3D2B,color:#DDDDDD
    style ATTR1 fill:#1D2E33,stroke:#205258,color:#DDDDDD
    style EVENTS1 fill:#312724,stroke:#5E3D2B,color:#DDDDDD
    style TC2_DETAILS fill:#212327,stroke:#3D4044,color:#DDDDDD
```

## Requirements

* **Gentrace SDK Initialization**: Must call [`init()`](../getting-started/initialization) with a valid API key. The SDK automatically configures OpenTelemetry for you. For custom OpenTelemetry setups, see the [manual setup guide](../reference/opentelemetry-guide)
* **Experiment Context**: Must be called within an [`experiment()`](./experiments) function
* **Valid Data Provider**: The `data` function must return an array of test cases

## Related functions

* [`init()`](../getting-started/initialization) - Initialize the Gentrace SDK
* [`interaction()`](../tracing/interactions) - Instrument AI functions for tracing within experiments
* [`experiment()`](./experiments) - Create experiment context for dataset evaluations
* [`eval()` / `evalOnce()`](./unit-tests) - Run individual test cases within an experiment
* [`traced()`](../tracing/traced) - Alternative approach for tracing functions