Agents fail differently than normal software.
You need traces and evals to find out why.
Every span. Understood.
Latency, inputs, outputs, tokens and cost, recorded per step, so a bad answer has a traceable cause.
Every score, explained.
Judges explain the score, so a low score points to exactly what went wrong.
Feedback Results
Metrics (click to learn more)
Don't take our word for it.
TruLens judges are best in class when graded against human annotations, out of the box.
Then tune them to your domain.
Start with state of the art judges, align judges to your domain with your private data. Add a rubric, examples, adjust the score range with the plumbing done for you.
from trulens.core import Metric, Selector
f_relevance = Metric(
implementation=provider.relevance_with_cot_reasons,
name="Answer Relevance",
selectors={
"prompt": Selector.select_record_input(),
"response": Selector.select_record_output(),
},
)f_relevance = Metric(
implementation=provider.relevance_with_cot_reasons,
name="Answer Relevance",
selectors={
"prompt": Selector.select_record_input(),
"response": Selector.select_record_output(),
},
# your rubric, replacing the shipped one
criteria="Score 3 only if the answer cites the policy section it used.",
additional_instructions="A section number with no quote is partial credit.",
# few-shot examples: (arguments, score)
examples=[
({"prompt": "Is water damage covered?",
"response": "Yes, section 4.2 covers burst pipes."}, 3),
({"prompt": "Is water damage covered?",
"response": "Yes, probably."}, 0),
],
# and the scale it scores on
min_score_val=0,
max_score_val=3,
)Compare versions.
Ship the frontier.
One tool call with the wrong arguments put the wrong policy in context and a wrong answer in front of a customer. Fixing it moved correctness 0.27 to 0.90. Additional experimentation unlocks further improvement and helps you find the quality/cost frontier.
What TruLens evaluates for you.
Agents
Did it pick the right tool and follow its plan?
- Tool Selection
- Plan Adherence
- Execution Efficiency
RAG
Is the answer supported by what was retrieved?
- Groundedness
- Context Relevance
- Answer Relevance
MCP apps
Are your MCP servers returning usable results?
- Tool Calling
- Tool Quality
- MCP span tracing
Summarization
Did it keep the substance and drop the noise?
- Comprehensiveness
- Groundedness
- Conciseness
OpenTelemetry In.
Traces Out.
Instrument any app in plain python, or auto-instrument your favorite framework.
from trulens.core.otel.instrument import instrument
from trulens.otel.semconv.trace import SpanAttributes
@instrument(
span_type=SpanAttributes.SpanType.RETRIEVAL,
attributes={
SpanAttributes.RETRIEVAL.RETRIEVED_CONTEXTS: "return",
},
)
def retrieve(self, query: str) -> list:
return self.vector_store.search(query)from trulens.apps.langgraph import TruGraph
app = TruGraph(
graph,
app_name="SupportAgent",
app_version="v2",
feedbacks=[f_tool_selection, f_plan_adherence],
)
with app as recording:
graph.invoke({"messages": [msg]})from trulens.apps.llamaindex import TruLlama
app = TruLlama(
query_engine,
app_name="DocsRAG",
app_version="v1",
feedbacks=[f_groundedness, f_context_relevance],
)
with app as recording:
query_engine.query("What changed in 2.8?")Live, or over a dataset.
from trulens.apps.app import TruApp
tru_rag = TruApp(
rag,
app_name="SupportAgent",
app_version="v3",
feedbacks=[f_groundedness, f_answer_relevance, f_context_relevance],
)
# recording a trace kicks off evaluation computation
with tru_rag as recording:
rag.query("Which policy covers this claim?")from trulens.core.run import RunConfig
run = tru_rag.add_run(RunConfig(
run_name="nightly-v3",
dataset_name="support_questions",
source_type="DATAFRAME",
dataset_spec={"input": "question"},
))
run.start(input_df=questions)
run.compute_metrics([f_groundedness, f_answer_relevance, f_context_relevance])
run.get_status() # RunStatus.COMPLETEDRecording a trace starts evaluation; a run replays a dataset. Same metrics either way, and the eval spans land in the trace, scores and leaderboard views above.
The breadth of evals
agentic apps actually need.
Agentic
- Tool Selection
- Tool Calling
- Tool Quality
- Plan Adherence
- Plan Quality
- Execution Efficiency
- Logical Consistency
Retrieval & RAG
- Context Relevance
- Groundedness
- Answer Relevance
- Comprehensiveness
Safety
- Harmfulness
- Toxicity
- Maliciousness
- Stereotyping
Quality
- Coherence
- Conciseness
- Sentiment
- Language match
Groundtruth
- Agreement
- Ranking (NDCG, MRR)
- Precision / Recall
Custom
- Any Python function
- Any LLM-as-judge prompt
- Any span attribute
Bring your own stack.
No rewrite, no lock-in. TruLens traces the app you built, judges with the model you chose, and writes to the database you run.
Frameworks & agentsInstrumented, span by span
Models & judgesSwap judges without rewriting an eval
RetrievalTraced and scored for relevance
Logging & platformsWhere the traces land
Get Started.
Quickstarts for LangChain, LangGraph, LlamaIndex and custom apps.
The largest eye in nature,
with its own headlights.
A colossal squid's eyeball is the size of a soccer ball, and it carries light organs that illuminate whatever it looks at. That is the job here: make the dark parts of your agent visible, then measure them.
Shepherded by Snowflake ↗
Originally created by TruEra and used by thousands of developers. Since the acquisition, Snowflake actively maintains TruLens in open source.