karwise/ evalsGuides & tools
AI EvalsFA
AI QUALITY, MADE UNDERSTANDABLE

What are AI Evals?
Make quality measurable.

AI evaluations check outputs against clear criteria.
Measure quality, catch mistakes, and compare changes with the same tests.

Facts for this exampleWay shoesDelivery: 3 daysReturns: 7 days
Before checkingGmail
Product email draft

Step into comfort with Way shoes.
Free instant shipping and 30-day returns!
View your favourite model.

Sounds nice. Wrong promises.2 of 3 checks passed
Check the answer
✓True promises?
✓Clear invitation?
✓Short and readable?
After fixing itGmail
Corrected email draft

Try Way shoes with 3-day delivery and a 7-day returns period.
View your favourite model.

Clear. Truthful. Ready to review.3 of 3 checks passed
A confident answer can still be wrong.

That is an eval. An example + your checks + a result you can explain.

Start with your own output. Examples are optional.

Changed your AI? Keep the same checks.

Try an ordinary example, a difficult one, and one that should fail. Compare the results before trusting the change.

TAKE EVALS INTO YOUR REAL WORK

Keep testing as your system changes.

Use datasets, run histories and repeated experiments to evaluate an agent or workflow over time. These three tools help you build that practice.

LangSmith

Datasets · Experiments · Traces

Compare versions on the same dataset and inspect the runs behind each score.

Langfuse

Production traces · Feedback · Judges

Score live traces, add human feedback, and turn useful cases into repeatable experiments.

Promptfoo

Code checks · Rubrics · Regression tests

Run code assertions and LLM rubric checks repeatedly as prompts, models or code change.

Bring your criteria and test cases into the tool you choose. The download uses the karwise format; it is not a one-click import for these platforms. Calibrate AI judges against human-reviewed examples.