- Check that the output contains a whitelist (or blacklist) of keywords
- Check that the output only contains alphanumeric characters
- Validate that the output is in JSON format
These heuristic functions are passed a single
attributes object. You can destructure any of the following keys from the object to compute your evaluation.
Test case values
inputsis an object that represents all the key/value pairs that were defined on the "inputs" value of a test case
expectedOutputsis an object that represents all of the key/value pairs that that were defined on the "expected outputs" value from a test case
Test result values
outputsis an object with the actual output key/value pairs from a test run of your generative pipeline
stepsreferences the list of output steps that is created by your generative pipeline.
Testing heuristic evaluators
Gentrace provides an interactive tool to check that your heuristic evaluator works correctly. Specify a test case, expected outputs, and test outputs to run the evaluation.