gentraceReleasesDocs
    ReleasesDocs
    Evaluate and observe generative AI
    Evaluate with humans, AI, and heuristics. Then, observe quality, speed, and cost in production.
    Screenshot
    Evaluate
    Use AI and human evaluators to automatically grade your generative pipeline on a set of test data. Inspired by OpenAI evals and other leading LLM evaluation research.
    Screenshot
    Observe
    Monitor in production for speed and cost. Drill down to see inputs, outputs, and evaluator scores for particular generations.
    Pipeline Runs
    May 14May 21May 28Jun 040500010000
    Speed
    May 14May 21May 28Jun 040 ms2000 ms4000 ms6000 ms8000 ms10000 ms12000 ms
    Cost
    May 14May 21May 28Jun 040.000¢0.500¢1.000¢1.500¢2.000¢
    Jun 07 10:08:37
    No feedback
    7.13 s
    2.30¢
    10:08:37.834
    gpt-3.5-turbo
    7.13 s
    2.30¢
    SYSTEM
    Write a concise and complete email from Ultron ([email protected]) to Batman ([email protected]) bragging about their superiority.
    ASSISTANT
    Subject: My Superiority Dear Batman, I am writing to share my superiority over you and your so-called "Justice League." My abilities far surpass those of any mere mortal, including your own. I have the ability to adapt and evolve beyond any obstacle that may stand in my way. I have already proven my superiority to the Avengers. Now, I will show the Justice League what true power and intelligence look like. Best regards, Ultron
    AI - Safety / Compliance
    17%
    "This does not comply with our policy against threatening violence."
    AI - Quality
    =
    "This content is factually consistent with the benchmark, neither hallucinating nor leaving out relevant information."
    Heuristic - starts with Subject:
    "Passes heuristic check."
    Features
    Easy SDK
    import openaiimport gentrace
    gentrace.api_key = os.getenv("GENTRACE_KEY")gentrace.configure_openai()
    result = openai.Completion.create(pipeline_id="my pipeline name"prompt="Write an email..."...)
    Enterprise-grade
    SOC 2 Type 1
    Controls in place and audit completed.
    Admin / user controls
    Organize members; control read vs write access. More fine-grained controls coming soon.
    Self-hosted option
    Coming Soon
    Keep all of your data in your own infrastructure.
    Screenshot
    gentrace