LLM-as-a-judge evaluation uses an LLM to grade an output from an AI system, augmenting or replacing manual, human evaluation.

However, when used for quality, LLM-as-a-judge evaluations frequently face significant skepticism:

: AI engineers ask "how can I use an LLM to judge something that was generated by an LLM, even potentially the same LLM, and actually get a quality outcome?"

: Or, they try using it, get bad results, and resign themselves to the fact that they will just need manual / human evaluations.

LLM-as-a-judge definitely can perform poorly. However, by giving the LLM an "unfair advantage" relative to the original task, I have seen that our customers are generally able to make the evaluation highly reliable.

In this post, I run through a bad evaluation, define unfair advantages, and then give concrete examples of several ways to give your evaluation an unfair advantage.

This advice applies if you're building a simple LLM feature, an agent, a Retrieval-Augmented Generation (RAG) pipeline, or any AI application that involves LLM-generated content.

Bad evaluations just ask the same question again

Let's provide a concrete example of a bad prompt that has no unfair advantages.

Consider the following, very simple LLM single-prompt feature which writes emails on behalf of a user.

An example of a bad LLM-as-a-judge evaluation would be:

This evaluation is bad because it asks the LLM the same question a second time - after you write an email given some information, grade the email you just wrote.

Is there any signal here? Not nearly enough to make changes to my source prompt / model with confidence. Let's try to create some unfair advantages to get better evaluation performance.

Better evaluations through "unfair advantages"

To make LLM-as-a-judge evaluations reliable, we need to give the LLM an "unfair advantage" during evaluation - some sort of additional context, tool, or capability that makes the evaluation task easier than the original generation task.

These unfair advantages help the LLM perform better by simplifying the evaluation into much more straightforward tasks with clearer criteria. This way, even smaller, faster models can give us high-confidence evaluations.

The best unfair advantage: tailored asserts

The absolute best way of building evaluations that have an extraordinarily unfair advantage is to use tailored asserts.

Then, I ask the LLM to grade the assert pass rate:

Pretty much any modern LLM will be able to handle this prompt well. I used Claude 3.5 haiku just to demonstrate how even a small model performs well.

If you can do some evaluations this way, even a relatively small number (e.g. 10 examples) will give you a lot more signal than a gigantic dataset with generic evals that don't have an unfair advantage.

To best operationalize this approach, make sure you iterate on your asserts while also iterating on your test data set and application. As you see new bad behaviors, add new asserts to guard against them and then go stamp them out.

The goal should still not be to get to 100% - that's almost impossible with LLMs - but to make more pass until users are delighted with the feature.

The next best: specific comparison to a reference output

The next best unfair advantage is to ask an LLM to do a comparison to a reference (where the reference is the unfair advantage).

Let's put a reference email into our email:

and create an evaluation that compares the content:

Now, the analysis is a bit more sophisticated but still relatively objective and easy.

In our case, Haiku isn't able to solve this, but latest Sonnet does well:

In any case, you will get a significantly higher quality answer than asking the equivalent non-comparative question, e.g. "is this verbose" or "is this clear."

Other unfair advantages to consider... or not

(Works well but situational) Use multi-modal advantages

Example: I'm generating a web component in response to a user query. 

Asking whether the generated HTML output is good for the query might not yield significant results. However, evaluating the image version can be more effective because the grader benefits from the visual representation during grading.

This approach only works when there is a multi-modal representation of the final output that is much easier to grade.

Note: this approach can be combined with asserts or comparison for very good results.

Rather than relying on specific asserts, you may want to create a general rubric of characteristics that should be present in the output.

Example: The output should not include any irrelevant information

However, these instructions generally can also be included in the original prompt, and therefore aren't really an "unfair" advantage. Evaluation reliability therefore suffers.

General rubrics can work well for evaluations that aren't geared around output quality - e.g., for measuring safety, a rubric of "here are the things we shouldn't ever talk about" can be very useful.

However, for quality, you're generally better off choosing another route.

(Frequently bad) Use a stronger model as the grader

This is highly situational - most times, asking a weaker model to generate and a stronger model to grade doesn't yield great results without another unfair advantage of some sort. Also, this comes with significant cost and slower grading.

However, when the task involves significant reasoning and the model is really weak, a higher level model can sometimes accurately reason through the answer.

Example: I'm generating feedback on a legal document

When generated using a weaker model, asking "is this feedback good" with a strong model (e.g. o1) may yield some useful results.

Always ask "how can I create an unfair advantage?"

Hopefully, this is helpful framing for you to consider as you build out your AI evaluations. The goal is to give you a framework more than a particular answer for your application - to always ask the question.

Gentrace is a tool for building evaluations (like these) and monitoring in test and production. Learn more about Gentrace 

LLM-as-a-judge evaluations become more reliable with an "unfair advantage" during evaluation.

Unfair advantages - a framework for building LLM-as-a-judge evaluations that reliably work

Quizlet’s goal: make learning faster and easier

Quizlet is a leading education technology company that provides online study tools for students and teachers.

To help students and teachers spend less time building materials and more time learning, Quizlet introduced their AI Study Era initiative, which aims to transform unstructured notes and materials into effective study tools (flash cards, study guides, practice tests) using generative AI.

However, as Quizlet developed these AI tools, they struggled with a lack of predictability. Even minor changes, such as adding a comma in a prompt, could significantly impact the downstream results.

From a home-grown testing stack to Gentrace

To systematically develop better generative AI features, Quizlet initially developed a homegrown evaluation setup, but the evaluation process was cumbersome. It relied on multiple Google Sheets and Colab notebooks, which made testing slow and inefficient.

Quizlet then chose Gentrace to address these challenges for several reasons:

: Gentrace allowed Quizlet to implement their own custom evaluations, which was crucial for their specific use cases.

: The platform provided easy visualization of results, allowing the team to dig into different views and quickly identify issues.

: Gentrace streamlined the testing process, making it faster and more accessible for any size of change.

Madeline Gilbert, Staff Machine Learning Engineer at Quizlet, explains:

"Gentrace was the right product for us because it allowed us to implement our own custom evaluations, which was crucial for our unique use cases. The ability to easily visualize what was going wrong and dig into the results with different types of views has been invaluable. It's dramatically improved our ability to predict the impact of even small changes in our LLM implementations."

Quizlet integrated Gentrace into their development and testing workflow:

: Before merging any changes, the team runs Gentrace to compare the main branch with the proposed changes, quantifying the impact.

: Quizlet uses a combination of AI (80%) and heuristic (20%) evaluators within Gentrace.

The implementation of Gentrace has led to significant improvements in Quizlet's AI development process:

: Reduced from approximately 1 hour to 1 minute or less.

: Better predictability and confidence in the impact of changes.

: Increased from 2 times per month to over 20 times per week.

: Significantly improved speed of iteration with testing.

With higher frequency testing and rapid iteration, Quizlet upgraded models with confidence and shipped higher quality outputs, allowing them to hit their deadline of launching significant improvements before the start of the 2024-2025 school year.

Quizlet increased the amount of testing by 40x and improved testing speed from 1hr -> 1 minute.

Quizlet builds automatic, high quality study materials with Gentrace

Gentrace suffered an evaluator outage that affected all customers who ran tests from Friday, July 26th at 1:42PM PT to Monday, July 29th at ~4:30 PM PT.

We did not automatically detect this issue pre-deploy or in our alerting post-deploy. We learned when a customer reported the issue on Monday at 3:17pm PT and we promoted to a P0.

Over the last few months, as users have submitted increasingly more data to Gentrace, we noticed that our databases (Clickhouse and Postgres) were struggling to keep up with the submissions which would often degrade page load times for end-users and cause frustrating evaluation delays.

A huge source of this load was due to our task runner. We only passed our evaluation tasks IDs to operate on, so they would refetch data and then perform the evaluation. Those refetches were adding up to a ton of load.

To fix this, we introduced a new architecture where all necessary data is included 

. This eliminated the excessive querying because data is included up front rather than re-fetched in the task.

This significantly improved performance but resulted in a critical bug where evaluators failed to pass outputs data into the evaluator context correctly. This resulted in evaluations failing because those values were blank.

Although we manually ran evaluations on our seed data, our testing focused on submitting AI evaluations to gain conviction that basic functionality was working. These AI evaluations were “silently failing” because our AI evaluations still returned the expected evaluation even when the prompt is malformed. Had we looked directly at the generated evaluation prompt, the issue would have been evident.

Our automated testing was built under the assumption that tasks fetch and process their own data. As such, we had tests similar to this:

, we changed our tests to reflect the new way we passed data.

The problem is the modified tests only tested that, given correctly formatted data, the evaluation was correct. It did not test whether data was being correctly formatted, which is where the bug occurred.

Our monitoring differentiates evaluation failures and task failures. A task failure is where the task errors out, whereas an evaluation failure is shown to the user so that they can then fix their evaluator.

In this case, because we were interpolating blank data into the evaluators, our monitoring considered it to be an evaluation failure rather than a task failure.

Currently, we only alert on spikes in task failure rather than evaluation failure, as evaluation failure can happen for user-generated reasons. As such, we did not alert here.

To improve our automated testing, we plan to add “router-level” integration tests for test result submission that test the entire lifecycle of an evaluation rather than just a single evaluation task.

We will add anomaly detection and alerting for spikes in evaluation failure rates, not just >0 task failures, as spikes in evaluation failures can also indicate an issue on our end.

To improve our manual testing, improve the variety of examples in our seed data such that evaluator failures will be more obvious.

Validate existing task failure monitoring and alerting.

Incident 7/29/2024: Evaluator outage

TLDR: A XSS vulnerability has been patched. We investigated and found no known affected users or recommended actions for Gentrace users.

On Friday Jan 26th at 2:59 PM PT, a Gentrace customer reported that a security consultant for their company had discovered a cross-site scripting (XSS) vulnerability in Gentrace.

This vulnerability occurred when unsanitized text was rendered in the OpenAI inputs and outputs blocks in the Gentrace UI when rendering OpenAI LLM calls. This happened because we use Mustache to interpolate any contentInputs into the contentTemplate and/or content from our SDK. Mustache sanitizes the inputs, but does not sanitize the template automatically.

On Sunday Jan 28th at 11:57 AM PT, we deployed a fix for this issue by escaping the content / content template before rendering in Mustache.

We developed and ran scripts to scan our production database for XSS vectors. We did not get any hits outside of the customer who reported the issue, and the hits on that customer were found to be the benign examples conducted by the security researcher.

Out of an abundance of caution, we also audited the entire codebase to ensure that all content with a user-generated component passed to React's dangerouslySetInnerHTML is wrapped in a DOMPurify call, which will prevent accidental XSS issues from arising in the future. This includes the Mustache rendered OpenAI content and any other LLM content.

To make this part of our PR process, we also implemented the react/no-danger eslint error, which will prevent PR checks from passing when using React dangerouslySetHTML unless reviewed for safety.

Gentrace takes your security very safely. Please reach out to us if there are other best-practice actions you'd recommend that we take.

Because the vulnerability was fixed and no users were found to be affected, we do not recommend any actions be taken by our customers.

A Gentrace user detected a XSS vulnerability, which has since been fixed. No users were found to be affected.

Security advisory: XSS vulnerability patched

AI hallucinations occur when an LLM returns incoherent and/or factually inaccurate information in response to a query.

Hallucinations can be fixed through many types of iterative enhancements, including higher quality context, a more pointed prompt, fine-tuning, or post-processing of generative output.

Developers need to test these (and other) changes to see if they improve or worsen hallucination.

In this post, I share four strategies for testing for hallucination. Two are totally automated, one involves an expert, and one requires end-user feedback.

A fast general-purpose way to test for hallucination is to do an automated comparison of your generative AI's output to a task to one or more ground truths or expected values.

In order to perform automated factual comparison, first create a test dataset.

A good test dataset consists of an input and some ground truth. The ground truth is organized however you want, but I find the easiest way is to just use a single value representing an ideal response from the generative AI.

, a good test dataset might look like this:

Run your test dataset inputs through your generative AI feature, converting the inputs to outputs.

Then, grade the output using an automated comparison of factualness. To do this, use AI or model-graded evaluation. This technique prompts an AI and asks it to do the grading for us.

I recomend using the following prompt as a starting point (from the 

As you can see, this template returns a value between A and E. Let's dive into what each of them mean.

 - there is some fact(s) in the expected value that did not make it to the actual output

 - there are some "new facts" in the output that weren't present in the expected output

 - there is an actual discrepancy between the facts

 - there is no factual difference, but they do look different

B (and occasionally D) show possible hallucination. Measure how often they occur across your test dataset. Consider also tracking omission (A).

But wait, can I really trust a LLM to grade itself?

It's a good question; in this case, the answer is generally yes.

To give a feel for why, consider how the LLM originally did a very difficult thing - produce something from scratch.

In this evaluation, the LLM does something much easier - compare two things.

As a result, even if the same model was used to produce and evaluate the output, it will do a much better job of finding factual mismatches than of producing the result.

Consider the case of this email generator.

Jack Matthews is never mentioned as a Stanford alumni, but GPT-3.5, attempting to write the best email possible, decides to slip it in.

Let's say this is happening frequently in your email generator; way more than any other form of hallucination.

Method 1 is overkill in this case. We just need to create an evaluator that looks out for this particular form of hallucination.

As in Method 1, first produce a test dataset. This time, there's no need for expected values.

Once you convert the inputs into outputs, again leverage model-graded evaluation, only this time with a more specific prompt:

The number of (A) answers are the amount of this specific type of hallucination.

Note that this form of automated evaluation has a nice characteristic - it doesn't rely on ground truth / an expected value, so it can be run against production traffic as it happens.

Consider siphoning ~100 samples per day from your production traffic to specific evaluations like this to measure production hallucination.

The third method is to have experts manually review generations for hallucination.

To make this work correctly, make sure that experts have acess to all information. It should be easy for the expert to validate / invalidate facts if they don't have the information in their head.

This approach is slow as it requires manual human intervention.

To make things faster, pair this method with the automated methods above. For example, you may want to run automated checks on 100% of test outputs but only manually review 10% of them.

The last way to measure hallucination is to provide end-users with a way to flag it.

An example: you implement a thumbs up / thumbs down UI that shows up whenever a generation is presented to the user. When the user clicks the thumbs down, the UI asks why the generation was wrong, where hallucination is a possible option.

The data isn't perfect. End-users flag hallucinations that aren't actually there, or more frequently, they miss hallucinations or fail to report them.

Even so, the data is directionally useful as this "hallucination report rate" can be used a benchmark for a (likely higher) true hallucination rate.

Additionally, the data can be improved by combining this mechanism with other methods above; e.g. you could have an expert review the potential hallucinations to remove false positives.

There's no one-size-fits-all solution to the hallucination problem. Start with automated checks, and then experiment with approaches in both development and production to maximize your odds of success.

This post covers four main ways to test for AI hallucination: automated comparison to truths, specific hallucination tests, expert review, and end-user review.

How to test for AI hallucination

One of the most common patterns in modern LLM development is Retrieval-Augmented Generation (RAG).

These systems respond to queries by retrieving relevant information from data sources and then generating a response to the query with an LLM using that information.

Note that in this post I use a broad definition of RAG which includes keyword-based RAG and other RAG architectures. Some people use a more scoped-down definition that requires embeddings and a Vector DB for the retrieval step.

, which uses RAG to query the public internet and an OpenAI model to generate a response to a query

, which query your knowledge base to generate a response

, which creates a high-level summary of all customer feedback about a product

For generative AI developers, RAG can be an easy way to stand up a quick MVP (“Wow, look at how our customers can now chat with their data in our product!”). But, once they get something up and running, developers struggle to systematically make changes to improve quality and reduce hallucination.

While it’s still a new field, best practices for testing RAG systems have already emerged and can be gradually implemented as follows.

In any of the scenarios below, you’ll need some test data.

Generally, a good test system consists of the following:

Access to a large data system on which you’ll be doing retrieval

A suite of 10+ test queries and good responses, tagged with their user ID or any other metadata necessary to handle the query.

For example, let’s say we wanted to implement a Q&A for customers of our KB product.

Basic testing system: ask GPT-4 if the facts are right

Once you have your test suite, ask a model (eg GPT-4) if the facts are right. This technique, known as AI evaluation, requires some work to get right.

Frst, enrich your test dataset with good expected answers:

Then, programatically generate outputs using your RAG pipeline, and grade them using a prompt similar to the following.

This prompt converts the input query, expected value, and actual output into a grade between A and E.

Once you have automated grading working, compare and contrast different experiments to systematically get better performance.

With the basic pipeline, you’ll probably notice that it’s difficult to differentiate “retrieval” system failures and “generation” system failures.

Furthermore, the “Fact” evaluator does not do a good job of understanding the completeness of the response.

Let’s upgrade our testing system to the below:

Enhancement #1: Test specific stages of the RAG pipeline

This enhancement solves the issue of “where did my system break.”

To understand where failures are happening at-a-glance, extend your automated output collection to collect the outputs of each step.

Then, setup individual evaluations on each step.

For example, for the retrieval step you may want to use Ragas’ 

 evaluations to see how “good” your retrieval is performing.

And if you’re dealing with more complex RAG scenarios, the two main steps may have sub-steps. For example, your retrieval step might consist of one sub-step for SQL query generation and one for query execution. Test each sub-step as necessary.

Enhancement #2: break down the Fact evaluation into pieces

The “fact” evaluation above is a good starting point for RAG evaluation.

However, if you need to improve performance, I recommend breaking it down into a pair of evaluations.

This evaluation fails when there are new or contradictory facts in a generated output.

This evaluation fails when the generated output is not as complete in it’s response as the expected.

I’m Doug Safreno, co-founder and CEO at 

. Gentrace helps automate test and production evaluations like the ones mentioned above for technology companies leveraging generative AI. It is useful to improve quality of LLM outputs at scale.

Thank you to our customers and partners for helping me with this post. Also, thank you to the OpenAI Evals contributors and Ragas contributors for providing inspiration for many of the ideas.

Best practices for LLM developers testing Retrieval-Augmented Generation (RAG) systems.

How to test RAG systems

Over the course of 2023, you helped Gentrace grow from an internal tool to the generative AI evaluation platform for a number of technology companies building with generative AI.

We now have hundreds of active users, millions of evaluated data points, and many important production LLM products built with Gentrace (we hope to share more about which ones in the new year).

As the year closes out, we want to share a few pieces of news with you.

First, Yongxing Deng (prev. co-founder / CTO @ Aloft, Head of Experimentation Platform @ Slack) has joined Gentrace working on product and engineering.

Second, we are hiring exceptional founding engineers. Come define a safer and more reliable world in the age of generative AI. 

Third, customers, founders, and leaders have joined Gentrace in an angel-only fundraise, bringing our total funding to $6.25M. 

Tuomas Artman & Cristina Cordova – Co-founder, CTO & COO @ Linear

Cai GoGwilt – Co-founder, Chief Architect @ Ironclad

Garrett Lord – Co-founder, CEO @ Handshake

Martin Mao – Co-founder, CEO @ Chronosphere

Steve Bartel & Nick Bushak – Co-founders, CEO/CTO @ Gem

Sumeet Gajri – CSO @ Instabase, ex-Carta

Alex Atallah – Co-founder, ex-CTO @ Opensea

Nikhil Viswanathan & Joe Lau – Co-founders, CEO/CTO @ Alchemy

Jairam Ranganathan — CPO @ Motive (fka KeepTruckin)

Unfair advantages - a framework for building LLM-as-a-judge evaluations that reliably work

Quizlet builds automatic, high quality study materials with Gentrace

Incident 7/29/2024: Evaluator outage

Security advisory: XSS vulnerability patched

How to test for AI hallucination

How to test RAG systems

2023 in review

Company

Resources

Social