trulens.apps.langchain¶

trulens.apps.langchain ¶

Additional Dependency Required

To use this module, you must have the trulens-apps-langchain package installed.

pip install trulens-apps-langchain

Classes¶

WithFeedbackFilterDocuments ¶

Bases: VectorStoreRetriever

Attributes¶

threshold `instance-attribute` ¶

threshold: float

A VectorStoreRetriever that filters documents using a minimum threshold on a feedback function before returning them.

PARAMETER	DESCRIPTION
`feedback`	use this feedback function to score each document. TYPE: `Feedback`
`threshold`	and keep documents only if their feedback value is at least this threshold. TYPE: `float`

Example: "Using TruLens guardrail context filters with Langchain"

```python
from trulens.apps.langchain import WithFeedbackFilterDocuments

# note: feedback function used for guardrail must only return a score, not also reasons
feedback = Feedback(provider.context_relevance).on_input().on(context)

filtered_retriever = WithFeedbackFilterDocuments.of_retriever(
    retriever=retriever,
    feedback=feedback,
    threshold=0.5
)

rag_chain = {"context": filtered_retriever | format_docs, "question": RunnablePassthrough()} | prompt | llm | StrOutputParser()

tru_recorder = TruChain(rag_chain,
    app_name='ChatApplication',
    app_version='filtered_retriever',
)

with tru_recorder as recording:
    llm_response = rag_chain.invoke("What is Task Decomposition?")
```

Functions¶

of_retriever `staticmethod` ¶

of_retriever(
    retriever: VectorStoreRetriever, **kwargs: Any
)

Create a new instance of WithFeedbackFilterDocuments based on an existing retriever.

The new instance will:

Get relevant documents (like the existing retriever its based on).
Evaluate documents with a specified feedback function.
Filter out documents that do not meet the minimum threshold.

PARAMETER	DESCRIPTION
`retriever`	VectorStoreRetriever - the base retriever to use. TYPE: `VectorStoreRetriever`
`**kwargs`	additional keyword arguments. TYPE: `Any` DEFAULT: `{}`

Returns: - WithFeedbackFilterDocuments: a new instance of WithFeedbackFilterDocuments.

LangChainInstrument ¶

Bases: Instrument

Instrumentation for LangChain apps.

Attributes¶

INSTRUMENT `class-attribute` `instance-attribute` ¶

INSTRUMENT = '__tru_instrumented'

Attribute name to be used to flag instrumented objects/methods/others.

APPS `class-attribute` `instance-attribute` ¶

APPS = '__tru_apps'

Attribute name for storing apps that expect to be notified of calls.

Classes¶

Default ¶

Instrumentation specification for LangChain apps.

Attributes¶

MODULES class-attribute instance-attribute ¶

MODULES = {'langchain'}

Filter for module name prefix for modules to be instrumented.

CLASSES class-attribute instance-attribute ¶

CLASSES = lambda: {
    RunnableSerializable,
    Serializable,
    Document,
    Chain,
    BaseRetriever,
    BaseLLM,
    BasePromptTemplate,
    BaseMemory,
    BaseChatMemory,
    BaseChatMessageHistory,
    BaseSingleActionAgent,
    BaseMultiActionAgent,
    BaseLanguageModel,
    BaseTool,
    WithFeedbackFilterDocuments,
}

Filter for classes to be instrumented.

METHODS class-attribute instance-attribute ¶

METHODS: Dict[str, ClassFilter] = dict_set_with_multikey(
    {},
    {
        ("invoke", "ainvoke"): RunnableSerializable,
        ("save_context", "clear"): BaseMemory,
        (
            "run",
            "arun",
            "_call",
            "__call__",
            "_acall",
            "acall",
        ): Chain,
        (
            "_get_relevant_documents",
            "get_relevant_documents",
            "aget_relevant_documents",
            "_aget_relevant_documents",
            "invoke",
            "ainvoke",
        ): RunnableSerializable,
        ("plan", "aplan"): (
            BaseSingleActionAgent,
            BaseMultiActionAgent,
        ),
        ("_arun", "_run"): BaseTool,
    },
)

Methods to be instrumented.

Key is method name and value is filter for objects that need those methods instrumented

Functions¶

print_instrumentation ¶

print_instrumentation() -> None

Print out description of the modules, classes, methods this class will instrument.

to_instrument_object ¶

to_instrument_object(obj: object) -> bool

Determine whether the given object should be instrumented.

to_instrument_class ¶

to_instrument_class(cls: type) -> bool

Determine whether the given class should be instrumented.

to_instrument_module ¶

to_instrument_module(module_name: str) -> bool

Determine whether a module with the given (full) name should be instrumented.

tracked_method_wrapper ¶

tracked_method_wrapper(
    query: Lens,
    func: Callable,
    method_name: str,
    cls: type,
    obj: object,
)

Wrap a method to capture its inputs/outputs/errors.

instrument_method ¶

instrument_method(method_name: str, obj: Any, query: Lens)

Instrument a method.

instrument_class ¶

instrument_class(cls)

Instrument the given class cls's new method.

This is done so we can be aware when new instances are created and is needed for wrapped methods that dynamically create instances of classes we wish to instrument. As they will not be visible at the time we wrap the app, we need to pay attention to new to make a note of them when they are created and the creator's path. This path will be used to place these new instances in the app json structure.

instrument_object ¶

instrument_object(
    obj, query: Lens, done: Optional[Set[int]] = None
)

Instrument the given object obj and its components.

instrument_bound_methods ¶

instrument_bound_methods(obj: object, query: Lens)

Instrument functions that may be bound methods.

Some apps include either anonymous functions or manipulates methods that have self bound already. Our other instrumentation cannot handle those cases.

Warning

Experimental work in progress.

TruChain ¶

Bases: App

Recorder for LangChain applications.

This recorder is designed for LangChain apps, providing a way to instrument, log, and evaluate their behavior.

Example: "Creating a LangChain RAG application"

Consider an example LangChain RAG application. For the complete code
example, see [LangChain
Quickstart](https://www.trulens.org/trulens/getting_started/quickstarts/langchain_quickstart/).

```python
from langchain import hub
from langchain.chat_models import ChatOpenAI
from langchain.schema import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

retriever = vectorstore.as_retriever()

prompt = hub.pull("rlm/rag-prompt")
llm = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0)

rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)
```

Feedback functions can utilize the specific context produced by the application's retriever. This is achieved using the select_context method, which then can be used by a feedback selector, such as on(context).

Example: "Defining a feedback function"

```python
from trulens.providers.openai import OpenAI
from trulens.core import Feedback
import numpy as np

# Select context to be used in feedback.
from trulens.apps.langchain import TruChain
context = TruChain.select_context(rag_chain)


# Use feedback
f_context_relevance = (
    Feedback(provider.context_relevance_with_context_reasons)
    .on_input()
    .on(context)  # Refers to context defined from `select_context`
    .aggregate(np.mean)
)
```

The application can be wrapped in a TruChain recorder to provide logging and evaluation upon the application's use.

Example: "Using the TruChain recorder"

```python
from trulens.apps.langchain import TruChain

# Wrap application
tru_recorder = TruChain(
    chain,
    app_name="ChatApplication",
    app_version="chain_v1",
    feedbacks=[f_context_relevance]
)

# Record application runs
with tru_recorder as recording:
    chain("What is langchain?")
```

Further information about LangChain apps can be found on the LangChain Documentation page.

PARAMETER	DESCRIPTION
`app`	A LangChain application. TYPE: `Chain`
`**kwargs`	Additional arguments to pass to App and AppDefinition. TYPE: `dict` DEFAULT: `{}`

Attributes¶

tru_class_info `instance-attribute` ¶

tru_class_info: Class

Class information of this pydantic object for use in deserialization.

Using this odd key to not pollute attribute names in whatever class we mix this into. Should be the same as CLASS_INFO.

app_id `class-attribute` `instance-attribute` ¶

app_id: AppID = Field(frozen=True)

Unique identifier for this app.

Computed deterministically from app_name and app_version. Leaving it here for it to be dumped when serializing. Also making it read-only as it should not be changed after creation.

app_name `instance-attribute` ¶

app_name: AppName

Name for this app. Default is "default_app".

app_version `instance-attribute` ¶

app_version: AppVersion

Version tag for this app. Default is "base".

tags `instance-attribute` ¶

tags: Tags = tags

Tags for the app.

metadata `instance-attribute` ¶

metadata: Metadata

Metadata for the app.

feedback_definitions `class-attribute` `instance-attribute` ¶

feedback_definitions: Sequence[FeedbackDefinitionID] = []

Feedback functions to evaluate on each record.

feedback_mode `class-attribute` `instance-attribute` ¶

feedback_mode: FeedbackMode = WITH_APP_THREAD

How to evaluate feedback functions upon producing a record.

record_ingest_mode `instance-attribute` ¶

record_ingest_mode: RecordIngestMode = record_ingest_mode

Mode of records ingestion.

root_class `instance-attribute` ¶

root_class: Class

Class of the main instrumented object.

Ideally this would be a ClassVar but since we want to check this without instantiating the subclass of AppDefinition that would define it, we cannot use ClassVar.

initial_app_loader_dump `class-attribute` `instance-attribute` ¶

initial_app_loader_dump: Optional[SerialBytes] = None

Serialization of a function that loads an app.

Dump is of the initial app state before any invocations. This can be used to create a new session.

Warning

Experimental work in progress.

app_extra_json `instance-attribute` ¶

app_extra_json: JSON

Info to store about the app and to display in dashboard.

This can be used even if app itself cannot be serialized. app_extra_json, then, can stand in place for whatever data the user might want to keep track of about the app.

feedbacks `class-attribute` `instance-attribute` ¶

feedbacks: List[Feedback] = Field(
    exclude=True, default_factory=list
)

Feedback functions to evaluate on each record.

connector `class-attribute` `instance-attribute` ¶

connector: DBConnector = Field(
    default_factory=lambda: connector, exclude=True
)

Database connector.

If this is not provided, a DefaultDBConnector will be made (if not already) and used.

instrument `class-attribute` `instance-attribute` ¶

instrument: Optional[Instrument] = Field(None, exclude=True)

Instrumentation class.

This is needed for serialization as it tells us which objects we want to be included in the json representation of this app.

recording_contexts `class-attribute` `instance-attribute` ¶

recording_contexts: ContextVar[RecordingContext] = Field(
    None, exclude=True
)

Sequences of records produced by the this class used as a context manager are stored in a RecordingContext.

Using a context var so that context managers can be nested.

instrumented_methods `class-attribute` `instance-attribute` ¶

instrumented_methods: Dict[int, Dict[Callable, Lens]] = (
    Field(exclude=True, default_factory=dict)
)

Mapping of instrumented methods (by id(.) of owner object and the function) to their path in this app.

records_with_pending_feedback_results `class-attribute` `instance-attribute` ¶

records_with_pending_feedback_results: BlockingSet[
    Record
] = Field(exclude=True, default_factory=BlockingSet)

Records produced by this app which might have yet to finish feedback runs.

manage_pending_feedback_results_thread `class-attribute` `instance-attribute` ¶

manage_pending_feedback_results_thread: Optional[Thread] = (
    Field(exclude=True, default=None)
)

Thread for manager of pending feedback results queue.

See _manage_pending_feedback_results.

selector_check_warning `class-attribute` `instance-attribute` ¶

selector_check_warning: bool = False

Issue warnings when selectors are not found in the app with a placeholder record.

If False, constructor will raise an error instead.

selector_nocheck `class-attribute` `instance-attribute` ¶

selector_nocheck: bool = False

Ignore selector checks entirely.

This may be necessary if the expected record content cannot be determined before it is produced.

app `instance-attribute` ¶

app: Any

The langchain app to be instrumented.

root_callable `class-attribute` ¶

root_callable: FunctionOrMethod = Field(
    default_factory=lambda: of_callable(_call)
)

The root callable of the wrapped app.

Functions¶

on_method_instrumented ¶

on_method_instrumented(
    obj: object, func: Callable, path: Lens
)

Called by instrumentation system for every function requested to be instrumented by this app.

get_method_path ¶

get_method_path(obj: object, func: Callable) -> Lens

Get the path of the instrumented function method relative to this app.

get_methods_for_func ¶

get_methods_for_func(
    func: Callable,
) -> Iterable[Tuple[int, Callable, Lens]]

Get the methods (rather the inner functions) matching the given func and the path of each.

See WithInstrumentCallbacks.get_methods_for_func.

on_new_record ¶

on_new_record(func) -> Iterable[RecordingContext]

Called at the start of record creation.

See WithInstrumentCallbacks.on_new_record.

on_add_record ¶

on_add_record(
    ctx: RecordingContext,
    func: Callable,
    sig: Signature,
    bindings: BoundArguments,
    ret: Any,
    error: Any,
    perf: Perf,
    cost: Cost,
    existing_record: Optional[Record] = None,
) -> Record

Called by instrumented methods if they use _new_record to construct a record call list.

See WithInstrumentCallbacks.on_add_record.

__rich_repr__ ¶

__rich_repr__() -> Result

Requirement for pretty printing using the rich package.

load `staticmethod` ¶

load(obj, *args, **kwargs)

Deserialize/load this object using the class information in tru_class_info to lookup the actual class that will do the deserialization.

model_validate `classmethod` ¶

model_validate(*args, **kwargs) -> Any

Deserialized a jsonized version of the app into the instance of the class it was serialized from.

Note

This process uses extra information stored in the jsonized object and handled by WithClassInfo.

continue_session `staticmethod` ¶

continue_session(
    app_definition_json: JSON, app: Any
) -> AppDefinition

Instantiate the given app with the given state app_definition_json.

Warning

This is an experimental feature with ongoing work.

PARAMETER	DESCRIPTION
`app_definition_json`	The json serialized app. TYPE: `JSON`
`app`	The app to continue the session with. TYPE: `Any`

RETURNS	DESCRIPTION
`AppDefinition`	A new `AppDefinition` instance with the given `app` and the given `app_definition_json` state.

new_session `staticmethod` ¶

new_session(
    app_definition_json: JSON,
    initial_app_loader: Optional[Callable] = None,
) -> AppDefinition

Create an app instance at the start of a session.

Warning

This is an experimental feature with ongoing work.

Create a copy of the json serialized app with the enclosed app being initialized to its initial state before any records are produced (i.e. blank memory).

get_loadable_apps `staticmethod` ¶

get_loadable_apps()

Gets a list of all of the loadable apps.

Warning

This is an experimental feature with ongoing work.

This is those that have initial_app_loader_dump set.

select_inputs `classmethod` ¶

select_inputs() -> Lens

Get the path to the main app's call inputs.

select_outputs `classmethod` ¶

select_outputs() -> Lens

Get the path to the main app's call outputs.

wait_for_feedback_results ¶

wait_for_feedback_results(
    feedback_timeout: Optional[float] = None,
) -> List[Record]

Wait for all feedbacks functions to complete.

PARAMETER	DESCRIPTION
`feedback_timeout`	Timeout in seconds for waiting for feedback results for each feedback function. Note that this is not the total timeout for this entire blocking call. TYPE: `Optional[float]` DEFAULT: `None`

RETURNS	DESCRIPTION
`List[Record]`	A list of records that have been waited on. Note a record will be included even if a feedback computation for it failed or timed out.

This applies to all feedbacks on all records produced by this app. This call will block until finished and if new records are produced while this is running, it will include them.

json ¶

json(*args, **kwargs)

Create a json string representation of this app.

awith_ `async` ¶

awith_(
    func: CallableMaybeAwaitable[A, T], *args, **kwargs
) -> T

Call the given async func with the given *args and **kwargs while recording, producing func results. The record of the computation is available through other means like the database or dashboard. If you need a record of this execution immediately, you can use awith_record or the App as a context manager instead.

with_ `async` ¶

with_(func: Callable[[A], T], *args, **kwargs) -> T

Call the given async func with the given *args and **kwargs while recording, producing func results. The record of the computation is available through other means like the database or dashboard. If you need a record of this execution immediately, you can use awith_record or the App as a context manager instead.

with_record ¶

with_record(
    func: Callable[[A], T],
    *args,
    record_metadata: JSON = None,
    **kwargs
) -> Tuple[T, Record]

Call the given func with the given *args and **kwargs, producing its results as well as a record of the execution.

awith_record `async` ¶

awith_record(
    func: Callable[[A], Awaitable[T]],
    *args,
    record_metadata: JSON = None,
    **kwargs
) -> Tuple[T, Record]

Call the given func with the given *args and **kwargs, producing its results as well as a record of the execution.

dummy_record ¶

dummy_record(
    cost: Cost = mod_base_schema.Cost(),
    perf: Perf = mod_base_schema.Perf.now(),
    ts: datetime = datetime.datetime.now(),
    main_input: str = "main_input are strings.",
    main_output: str = "main_output are strings.",
    main_error: str = "main_error are strings.",
    meta: Dict = {"metakey": "meta are dicts"},
    tags: str = "tags are strings",
) -> Record

Create a dummy record with some of the expected structure without actually invoking the app.

The record is a guess of what an actual record might look like but will be missing information that can only be determined after a call is made.

All args are Record fields except these:

- `record_id` is generated using the default id naming schema.
- `app_id` is taken from this recorder.
- `calls` field is constructed based on instrumented methods.

instrumented ¶

instrumented() -> Iterable[Tuple[Lens, ComponentView]]

Iteration over instrumented components and their categories.

print_instrumented ¶

print_instrumented() -> None

Print the instrumented components and methods.

format_instrumented_methods ¶

format_instrumented_methods() -> str

Build a string containing a listing of instrumented methods.

print_instrumented_methods ¶

print_instrumented_methods() -> None

Print instrumented methods.

print_instrumented_components ¶

print_instrumented_components() -> None

Print instrumented components and their categories.

select_context `classmethod` ¶

select_context(app: Optional[Chain] = None) -> Lens

Get the path to the context in the query output.

main_input ¶

main_input(
    func: Callable, sig: Signature, bindings: BoundArguments
) -> str

Determine the main input string for the given function func with signature sig if it is to be called with the given bindings bindings.

main_output ¶

main_output(
    func: Callable,
    sig: Signature,
    bindings: BoundArguments,
    ret: Any,
) -> str

Determine the main out string for the given function func with signature sig after it is called with the given bindings and has returned ret.

acall_with_record `async` ¶

acall_with_record(*args, **kwargs) -> None

DEPRECATED: Run the chain acall method and also return a record metadata object.

call_with_record ¶

call_with_record(*args, **kwargs) -> None

DEPRECATED: Run the chain call method and also return a record metadata object.

call ¶

__call__(*args, **kwargs) -> None

DEPRECATED: Wrapped call to self.app._call with instrumentation. If you need to get the record, use call_with_record instead.

trulens.apps.langchain¶