> ## Documentation Index
> Fetch the complete documentation index at: https://next.gentrace.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Experiments

> Create and submit experiments to Gentrace with `experiment()`

An experiment in Gentrace is a group of traces created in an agent testing context along with associated [error analysis](/error-analysis/overview).

In this guide, we'll explain how to submit an experiment to Gentrace.

## Basic usage

Use the `experiment()` function to submit an experiment to Gentrace. It manages the lifecycle of a Gentrace experiment, automatically starting and finishing the experiment while providing context for evaluation functions like [`eval()` / `evalOnce()`](./unit-tests) and [`evalDataset()` / `eval_dataset()`](./dataset-tests).

<Note>
  **Important**: The `experiment()` function is designed to work with
  evaluation functions. To fully understand how to use experiments
  effectively, you should also review: - [`eval()` /
  `evalOnce()`](./unit-tests) - For running individual test cases -
  [`evalDataset()` / `eval_dataset()`](./dataset-tests) - For batch
  evaluation against datasets These functions must be called within an
  experiment context to properly track and group your test results.
</Note>

<CodeGroup dropdown>
  ```typescript theme={null}
  import { init, experiment, evalOnce } from 'gentrace';

  init({
    apiKey: process.env.GENTRACE_API_KEY,
  });

  const PIPELINE_ID = process.env.GENTRACE_PIPELINE_ID!;

  // Basic experiment usage
  experiment(PIPELINE_ID, async () => {
    await evalOnce('simple-test', async () => {
      // Your test logic here
      return 'test result';
    });
  });
  ```

  ```python theme={null}
  import os
  import asyncio
  from gentrace import init, experiment, eval
  init(api_key=os.environ["GENTRACE_API_KEY"])

  PIPELINE_ID = os.environ["GENTRACE_PIPELINE_ID"]

  @experiment(pipeline_id=PIPELINE_ID)
  async def my_experiment() -> None:
      @eval(name="simple-test")
      async def simple_test() -> str:
          # Your test logic here
          return "test result"

      await simple_test()

  # Run the experiment
  asyncio.run(my_experiment())
  ```
</CodeGroup>

<Note>
  When the `pipeline_id` parameter is omitted, the experiment will automatically submit to the default pipeline. In Python, use `@experiment` without parentheses when no parameters are provided. In TypeScript, use `experiment(async () => { ... })`. This is convenient for quick testing, but we recommend explicitly specifying the pipeline ID for production use.
</Note>

## Overview

An experiment in Gentrace represents a collection of test cases or evaluations run against your AI pipeline. The `experiment()` function:

1. **Creates an experiment run** in Gentrace with a unique experiment ID
2. **Provides context** for evaluation functions to associate their results with the experiment
3. **Manages lifecycle** by automatically starting and finishing the experiment
4. **Captures metadata** and organizes test results for analysis

## Parameters

<CodeGroup dropdown>
  ```typescript theme={null}
  /**
   * @param {string} pipelineId - The UUID of the Gentrace pipeline to
     associate with this experiment. When omitted, the experiment will
     automatically submit to the default pipeline
   * @param {() => T | Promise<T>} callback - The function containing
     your experiment logic and test cases
   * @param {ExperimentOptions} [options] - Additional config options
   * @param {Record<string, any>} [options.metadata] - Custom metadata
     to associate with the experiment run
   * @returns {Promise<T>}
   */
  function experiment<T>(
    pipelineId: string,
    callback: () => T | Promise<T>,
    options?: ExperimentOptions
  ): Promise<T>
  ```

  ```python theme={null}
  def experiment(
      *,
      pipeline_id: str,
      options: Optional[ExperimentOptions] = None,
  ) -> Callable[[Callable[P, Any]], Callable[P, Coroutine[Any, Any, None]]]:
      """
      Args:
          pipeline_id (str): The UUID of the Gentrace pipeline to
              associate with this experiment. When omitted, the experiment
              will automatically submit to the default pipeline
          options (ExperimentOptions, optional): Additional config options
          options.name (str, optional): A descriptive name for the
              experiment run
          options.metadata (Dict[str, Any], optional): Custom metadata
              to associate with the experiment run
      """
  ```
</CodeGroup>

## Advanced usage

### With metadata

<CodeGroup dropdown>
  ```typescript theme={null}
  import { experiment, evalOnce } from 'gentrace';

  experiment(
    PIPELINE_ID,
    async () => {
      await evalOnce('model-comparison-test', async () => {
        // Test logic comparing different models
        return { accuracy: 0.95, latency: 120 };
      });
    },
    {
      metadata: {
        model: 'o3',
        version: '1.2.0',
        environment: 'staging',
      },
    },
  );
  ```

  ```python theme={null}
  @experiment(
      pipeline_id=PIPELINE_ID,
      options={
          "name": "Model Comparison Experiment",
          "metadata": {
              "model": "o3",
              "version": "1.2.0",
              "environment": "staging"
          }
      }
  )
  async def model_comparison_experiment() -> None:
      @eval(name="model-comparison-test")
      async def model_comparison_test() -> dict:
          # Test logic comparing different models
          return {"accuracy": 0.95, "latency": 120}

      await model_comparison_test()
  ```
</CodeGroup>

### Multiple test cases

<CodeGroup dropdown>
  ```typescript theme={null}
  import {
    experiment,
    evalOnce,
    evalDataset,
    testCases,
  } from 'gentrace';
  experiment(PIPELINE_ID, async () => {
    // Individual test cases
    await evalOnce('accuracy-test', async () => {
      const result = await myAIFunction('test input');
      return { accuracy: calculateAccuracy(result) };
    });

    await evalOnce('latency-test', async () => {
      const start = Date.now();
      await myAIFunction('test input');
      const latency = Date.now() - start;
      return { latency };
    });

    // Dataset evaluation
    await evalDataset({
      data: async () => {
        const DATASET_ID = process.env.GENTRACE_DATASET_ID!;
        const testCasesList = await testCases.list({
          datasetId: DATASET_ID,
        });
        return testCases.data;
      },
      interaction: myAIFunction,
    });
  });
  ```

  ```python theme={null}
  @experiment(pipeline_id=PIPELINE_ID)
  async def comprehensive_experiment() -> None:
      @eval(name="accuracy-test")
      async def accuracy_test() -> dict:
          result = await my_ai_function("test input")
          return {"accuracy": calculate_accuracy(result)}

      @eval(name="latency-test")
      async def latency_test() -> dict:
          start = time.time()
          await my_ai_function("test input")
          latency = time.time() - start
          return {"latency": latency}

      # Run individual tests
      await accuracy_test()
      await latency_test()

      # Run dataset evaluation
      from gentrace import test_cases

      await eval_dataset(
          data=lambda: test_cases.list(dataset_id=DATASET_ID).data,
          interaction=my_ai_function,
      )
  ```
</CodeGroup>

## Context and lifecycle

The `experiment()` function manages the experiment lifecycle automatically:

<Steps>
  <Step title="Start">
    Creates a new [experiment run](./experiments#create-an-experiment)
    in Gentrace
  </Step>

  <Step title="Context">
    Provides experiment context to nested [evaluation
    functions](./unit-tests)
  </Step>

  <Step title="Execution">
    Runs your experiment [callback/function](#basic-usage)
  </Step>

  <Step title="Finish">
    Marks the experiment as complete in Gentrace with [status
    updates](./experiments#update-an-experiment)
  </Step>
</Steps>

### Accessing experiment context

<CodeGroup dropdown>
  ```typescript theme={null}
  import { getCurrentExperimentContext } from 'gentrace';

  experiment(PIPELINE_ID, async () => {
    const context = getCurrentExperimentContext();
    console.log('Experiment ID:', context?.experimentId);
    console.log('Pipeline ID:', context?.pipelineId);

    // Your test logic here
  });
  ```

  ```python theme={null}
  from gentrace import get_current_experiment_context

  @experiment(pipeline_id=PIPELINE_ID)
  async def my_experiment() -> None:
      context = get_current_experiment_context()
      print(f"Experiment ID: {context['experiment_id']}")
      print(f"Pipeline ID: {context['pipeline_id']}")

      # Your test logic here
  ```
</CodeGroup>

## Error handling

The experiment function handles errors gracefully and **automatically associates all errors and exceptions with the [OpenTelemetry span](https://opentelemetry.io/docs/concepts/signals/traces/#spans)**. When an error occurs within an experiment or evaluation, it is captured as [span events](https://opentelemetry.io/docs/concepts/signals/traces/#span-events) and [attributes](https://opentelemetry.io/docs/concepts/signals/traces/#attributes), providing full traceability in your observability stack.

<CodeGroup dropdown>
  ```typescript theme={null}
  experiment(PIPELINE_ID, async () => {
    try {
      await evalOnce('test-that-might-fail', async () => {
        // Test logic that might throw an error
        // This error will be automatically captured in the OTEL span
        throw new Error('Test failed');
      });
    } catch (error) {
      console.log('Test failed as expected:', error.message);
      // The error is already recorded in the span with full stack trace
    }

    // Experiment will still finish properly
    // All error information is preserved in the OpenTelemetry trace
  });
  ```

  ```python theme={null}
  @experiment(pipeline_id=PIPELINE_ID)
  async def error_handling_experiment() -> None:
      @eval(name="test-that-might-fail")
      async def failing_test() -> None:
          # Test logic that might raise an error
          # This exception will be automatically captured in the OTEL span
          raise ValueError("Test failed")

      try:
          await failing_test()
      except ValueError as e:
          print(f"Test failed as expected: {e}")
          # The exception is already recorded in the span with full stack trace

      # Experiment will still finish properly
      # All error information is preserved in the OpenTelemetry trace
  ```
</CodeGroup>

### OTEL span error integration

When errors occur within experiments:

* **Automatic Capture**: All `Error` objects (TypeScript) and exceptions (Python) are automatically captured as [span events](https://opentelemetry.io/docs/concepts/signals/traces/#span-events)
* **Stack Traces**: Full stack traces are preserved in the [span attributes](https://opentelemetry.io/docs/concepts/signals/traces/#attributes) for debugging
* **Error Attributes**: Error messages, types, and metadata are recorded as span attributes
* **Span Status**: The [span status](https://opentelemetry.io/docs/concepts/signals/traces/#span-status) is automatically set to `ERROR` when unhandled exceptions occur

This integration ensures that failed experiments and evaluations are fully observable and debuggable through your OpenTelemetry-compatible monitoring tools.

## Best practices

### 1. Use descriptive names and metadata

```typescript theme={null}
// Good: Descriptive metadata
experiment(
  PIPELINE_ID,
  async () => {
    // tests...
  },
  {
    metadata: {
      model: 'o3',
      prompt_version: 'v2.1',
      test_suite: 'regression',
      branch: 'feature/new-prompts',
    },
  },
);
```

### 2. Group related tests

Organize related test cases within a single experiment:

```typescript theme={null}
experiment(PIPELINE_ID, async () => {
  // All accuracy-related tests
  await evalOnce('accuracy-basic', async () => {
    /* ... */
  });
  await evalOnce('accuracy-edge-cases', async () => {
    /* ... */
  });

  // All performance-related tests
  await evalOnce('latency-test', async () => {
    /* ... */
  });
  await evalOnce('throughput-test', async () => {
    /* ... */
  });
});
```

### 3. Handle async operations properly

<CodeGroup dropdown>
  ```typescript theme={null}
  // Ensure all async operations are awaited
  experiment(PIPELINE_ID, async () => {
    await evalOnce('test-1', async () => {
      /* ... */
    });
    await evalOnce('test-2', async () => {
      /* ... */
    });

    // Run tests in parallel if they're independent
    await Promise.all([
      evalOnce('parallel-test-1', async () => {
        /* ... */
      }),
      evalOnce('parallel-test-2', async () => {
        /* ... */
      }),
    ]);
  });
  ```

  ```python theme={null}
  @experiment(pipeline_id=PIPELINE_ID)
  async def async_experiment() -> None:
      @eval(name="test-1")
      async def test_1() -> None:
          # Test logic here
          pass

      @eval(name="test-2")
      async def test_2() -> None:
          # Test logic here
          pass

      @eval(name="parallel-test-1")
      async def parallel_test_1() -> None:
          # Test logic here
          pass

      @eval(name="parallel-test-2")
      async def parallel_test_2() -> None:
          # Test logic here
          pass

      # Ensure all async operations are awaited
      await test_1()
      await test_2()

      # Run tests in parallel if they're independent
      await asyncio.gather(
          parallel_test_1(),
          parallel_test_2(),
      )
  ```
</CodeGroup>

## Requirements

* **Gentrace SDK Initialization**: Must call [`init()`](../getting-started/initialization) with a valid API key. The SDK automatically configures OpenTelemetry for you. For custom OpenTelemetry setups, see the [manual setup guide](../reference/opentelemetry-guide)
* **Valid Pipeline ID**: Must provide a valid UUID for an existing Gentrace pipeline

## Related functions

* [`init()`](../getting-started/initialization) - Initialize the Gentrace SDK
* [`interaction()`](../tracing/interactions) - Instrument AI functions for tracing within experiments
* [`evalDataset()` / `eval_dataset()`](./dataset-tests) - Run tests against a dataset within an experiment
* [`evalOnce()` / `eval()`](./unit-tests) - Run individual test cases within an experiment
* [`traced()`](../tracing/traced) - Alternative approach for tracing functions

## Next steps

After submitting an experiment, setup evaluations using [derivations](/error-analysis/derivations).