Heuristic evaluators

You can define functions using JavaScript to assess quality. For example, you can:

• Check that the output contains a whitelist (or blacklist) of keywords
• Check that the output only contains alphanumeric characters
• Validate that the output is in JSON format

These heuristic functions are passed a single attributes object. You can destructure any of the following keys from the object to compute your evaluation.

Test case values

• inputs is an object that represents all the key/value pairs that were defined on the "inputs" value of a test case
• expectedOutputs is an object that represents all of the key/value pairs that that were defined on the "expected outputs" value from a test case

Test result values

• outputs is an object with the actual output key/value pairs from a test run of your generative pipeline

Sandboxed functions

These functions are executed in sandboxed WebAssembly QuickJS contexts. Functions cannot invoke network requests or access host system resources. ES2020 JavaScript language features are supported out of the box.

Testing heuristic evaluators

Gentrace provides an interactive tool to check that your heuristic evaluator works correctly. Specify a test case, expected outputs, and test outputs to run the evaluation.