Skip to main content
Version: 4.7.43

Custom Provider Evaluators

Custom provider evaluators allow you to use your own AI models as judges in Gentrace's evaluation infrastructure. This guide will walk you through setting up and using custom models for LLM-as-judge evaluations.

🔗 Quick access: Custom Providers Settings

Custom Providers Overview

Overview​

Custom provider evaluators enable you to:

  • Use your own AI models as evaluation judges
  • Control evaluation costs with your infrastructure
  • Maintain consistent evaluation standards across providers

API Standards​

When configuring your evaluator provider, select the API standard that matches your endpoint's structure:

Standards​

  • OpenAI: Chat completions with system/user/assistant roles and function calling
  • Anthropic: Claude-style message format and parameters
  • Azure OpenAI: OpenAI-compatible with Azure endpoints and auth
  • Gemini: Google's API format with Cloud authentication

Common Requirements​

All endpoints must support:

  • Model listing and discovery
  • Completion/chat endpoints
  • Response parsing
  • Temperature and token controls
  • Error handling

Setup​

1. Navigate to Custom Providers​

In your Gentrace dashboard, go to Settings → Custom Providers. This section allows you to configure providers that will be accessible for evaluations.

Custom Providers Modal

2. Add a New Provider​

Click the "Add your first provider" button to begin setup. You'll need to configure:

  • Provider Name: A unique identifier for your provider
  • Base URL: The endpoint where your provider service is hosted
  • API Standard: Select the standard your API follows from the dropdown
  • Authentication: Required credentials to access your provider

3. Configure Evaluation Models​

Select the models that will perform evaluations. These models will be available when creating AI evaluators:

First, discover available models from your endpoint: Model Discovery

Then configure your evaluation models: Parser and Reasoning Configuration

  • Reasoning Model: Used as the default model when creating LLM-as-judge evaluators
  • Parser Model: Used to convert unstructured responses into structured data

4. Testing and Validation​

After configuring your provider:

  1. Test the connection to ensure proper communication
  2. Verify model discovery is working correctly
  3. Run sample evaluations to confirm end-to-end functionality

Using Custom Evaluators​

Now that your provider is configured, you can start using it for evaluations:

  1. Create an AI evaluator and select your custom provider's model
  2. Use our evaluator templates or create your own prompts
  3. Run evaluations using our local evaluation guide

For examples and best practices of AI evaluator templates that work well with custom providers, see our evaluator documentation.

For technical support or questions about implementation, please contact our support team at [email protected].