Agent evals and experiments
Agent evals are a repeatable check of application behavior. You replace guesswork with scores — numeric, categorical, boolean, or text — on live traces and on dataset experiments, so you can catch regressions before you ship.
This page is for teams that need to measure quality in production and compare prompt, model, or code changes before release. If you want to score traffic today, start with Evaluate Production Traffic. To test a change on a fixed set of cases, start with Evaluate with Datasets.
Evaluation runs across most of the AI engineering loop: you score live traces, turn interesting examples into datasets, run experiments, and judge the results with manual or automated evaluators. It happens both online, on live production traces, and offline, before you ship a change.
Want to see it in action? Create a free account and explore Langfuse Evaluation in the interactive example project.
What you can do with evals
Teams use Langfuse evals to:
- Score live production traces with LLM-as-a-Judge or code evaluators
- Review edge cases in annotation queues and collect user feedback
- Compare prompt, model, or code changes with experiments on a dataset
- Block a deploy when an experiment regresses in CI
Getting started
You can evaluate both:
- live incoming traces to measure quality on production data and track trends over time.
- results from running tests on a pre-defined dataset, to make sure your changes are ready for production.
For how evaluators, scores, datasets, and experiments fit together, read Core Concepts.
If you're looking for another specific workflow, use the table below to find the right feature page:
| If you want to... | Use this Langfuse feature |
|---|---|
| Review and rate traces manually | Annotation Queues, Scores via UI |
| Collect feedback from your end users | User Feedback |
| Leave open-ended notes on traces | Text scores, Annotation Queues |
| Build a reusable set of test cases | Datasets |
| Compare prompt, model, or code changes side by side | Experiments via UI, Experiments via SDK, Experiments via OpenTelemetry |
| Block deploys on regressions | CI/CD experiments |
| Run deterministic checks | Code Evaluators |
| Automatically score live production traces | LLM-as-a-Judge, Scores via API/SDK |
| See how scores trend over time | Score Analytics, custom dashboards |
Already know what you're looking for? Browse Evaluation Methods and Experiments in the sidebar.
FAQ
What are agent evals?
Agent evals are a repeatable way to score application behavior — helpfulness, factuality, tone, safety, or any rubric you define — instead of relying on anecdotal reviews. In Langfuse, evaluation results are stored as scores on traces, observations, sessions, and experiment runs.
What is the difference between online and offline evaluation?
Online evaluation scores live production traces as they arrive, so you can watch quality trends and catch regressions in production. Offline evaluation runs your application against a fixed dataset before you ship, so you can compare prompt, model, or code changes side by side. Experiments are the offline workflow.
When should I use LLM-as-a-Judge vs a code evaluator?
Use a code evaluator for deterministic checks you can write as Python or TypeScript — format, required fields, keyword presence, or numeric thresholds. Use LLM-as-a-Judge when the judgment is qualitative, such as faithfulness, helpfulness, or whether a response followed a policy.
How do experiments relate to scores?
An experiment runs your task on a dataset and attaches scores to each item. Those scores are the same score objects you use on production traces, so you can compare an offline experiment to live quality using one data model.
GitHub Discussions
Last edited
