Importing external evaluations

You don't need to build your evaluation stack in Gentrace to use the platform. You can import evaluations that you build yourself. To import external evaluations, you do need to set up an external evaluator which receives the evaluations.

Then, you can import evaluations from a file or via the API / SDK.

info

External evaluations can be used in experimentation, test, or production.

info

If you want to use Gentrace's SDK for creating local evaluations, you can read the doc here about running local evaluations.

Setup: External evaluators

To build an external evaluator, navigate to the new Evaluator page, then click "Human / External" and pick a template (any works).

Then, customize scoring to match your external evaluator. Note: this must exactly match the scoring that your external evaluator returns.

For example, if your external evaluator returns A, B, C, or D, you should set up options scoring.

Alternatively, if your external evaluator returns a score of 0.0 to 1.0, you should use percentage scoring.

Import from JSON, JSONL, or CSV

Once you've set up your external evaluators, you can import evaluations from a file.

First, construct a test result in Gentrace the normal way.

Then, navigate to the result, press the ellipsis in the top right, and click the "Import evaluations" button.

Now, follow the prompts to import your evaluations. Gentrace supports JSON, JSONL, and CSV files.

Import via API / SDK

You can also import evaluations via the API or SDK. This also requires that you've set up one or more external evaluators.

First, construct a test result in Gentrace the normal way.

Then, use our evaluation bulk create method to create evaluations.

• SDK
• API