trulens.core.schema¶

trulens.core.schema ¶

Serializable Classes¶

Note: Only put classes which can be serialized in this module.

Classes with non-serializable variants¶

Many of the classes defined here extending serial.SerialModel are meant to be serialized into json. Most are extended with non-serialized fields in other files.

Serializable	Non-serializable
AppDefinition	App, Tru{Chain, Llama, ...}
FeedbackDefinition	Feedback

AppDefinition.app is the JSON-ized version of a wrapped app while App.app is the actual wrapped app. We can thus inspect the contents of a wrapped app without having to construct it. Additionally, JSONized objects like AppDefinition.app feature information about the encoded object types in the dictionary under the util.py:CLASS_INFO key.

Classes¶

AppDefinition ¶

Bases: WithClassInfo, SerialModel

Serialized fields of an app here whereas App contains non-serialized fields.

Attributes¶

tru_class_info `instance-attribute` ¶

tru_class_info: Class

Class information of this pydantic object for use in deserialization.

Using this odd key to not pollute attribute names in whatever class we mix this into. Should be the same as CLASS_INFO.

app_id `class-attribute` `instance-attribute` ¶

app_id: AppID = Field(frozen=True)

Unique identifier for this app.

Computed deterministically from app_name and app_version. Leaving it here for it to be dumped when serializing. Also making it read-only as it should not be changed after creation.

app_name `instance-attribute` ¶

app_name: AppName

Name for this app. Default is "default_app".

app_version `instance-attribute` ¶

app_version: AppVersion

Version tag for this app. Default is "base".

tags `instance-attribute` ¶

tags: Tags = tags

Tags for the app.

metadata `instance-attribute` ¶

metadata: Metadata

Metadata for the app.

feedback_definitions `class-attribute` `instance-attribute` ¶

feedback_definitions: Sequence[FeedbackDefinitionID] = []

Feedback functions to evaluate on each record.

feedback_mode `class-attribute` `instance-attribute` ¶

feedback_mode: FeedbackMode = WITH_APP_THREAD

How to evaluate feedback functions upon producing a record.

record_ingest_mode `instance-attribute` ¶

record_ingest_mode: RecordIngestMode = record_ingest_mode

Mode of records ingestion.

root_class `instance-attribute` ¶

root_class: Class

Class of the main instrumented object.

Ideally this would be a ClassVar but since we want to check this without instantiating the subclass of AppDefinition that would define it, we cannot use ClassVar.

root_callable `class-attribute` ¶

root_callable: FunctionOrMethod

App's main method.

This is to be filled in by subclass.

app `instance-attribute` ¶

app: JSONized[AppDefinition]

Wrapped app in jsonized form.

initial_app_loader_dump `class-attribute` `instance-attribute` ¶

initial_app_loader_dump: Optional[SerialBytes] = None

Serialization of a function that loads an app.

Dump is of the initial app state before any invocations. This can be used to create a new session.

Warning

Experimental work in progress.

app_extra_json `instance-attribute` ¶

app_extra_json: JSON

Info to store about the app and to display in dashboard.

This can be used even if app itself cannot be serialized. app_extra_json, then, can stand in place for whatever data the user might want to keep track of about the app.

Functions¶

__rich_repr__ ¶

__rich_repr__() -> Result

Requirement for pretty printing using the rich package.

load `staticmethod` ¶

load(obj, *args, **kwargs)

Deserialize/load this object using the class information in tru_class_info to lookup the actual class that will do the deserialization.

model_validate `classmethod` ¶

model_validate(*args, **kwargs) -> Any

Deserialized a jsonized version of the app into the instance of the class it was serialized from.

Note

This process uses extra information stored in the jsonized object and handled by WithClassInfo.

continue_session `staticmethod` ¶

continue_session(
    app_definition_json: JSON, app: Any
) -> AppDefinition

Instantiate the given app with the given state app_definition_json.

Warning

This is an experimental feature with ongoing work.

PARAMETER	DESCRIPTION
`app_definition_json`	The json serialized app. TYPE: `JSON`
`app`	The app to continue the session with. TYPE: `Any`

RETURNS	DESCRIPTION
`AppDefinition`	A new `AppDefinition` instance with the given `app` and the given `app_definition_json` state.

new_session `staticmethod` ¶

new_session(
    app_definition_json: JSON,
    initial_app_loader: Optional[Callable] = None,
) -> AppDefinition

Create an app instance at the start of a session.

Warning

This is an experimental feature with ongoing work.

Create a copy of the json serialized app with the enclosed app being initialized to its initial state before any records are produced (i.e. blank memory).

get_loadable_apps `staticmethod` ¶

get_loadable_apps()

Gets a list of all of the loadable apps.

Warning

This is an experimental feature with ongoing work.

This is those that have initial_app_loader_dump set.

select_inputs `classmethod` ¶

select_inputs() -> Lens

Get the path to the main app's call inputs.

select_outputs `classmethod` ¶

select_outputs() -> Lens

Get the path to the main app's call outputs.

FeedbackDefinition ¶

Bases: WithClassInfo, SerialModel, Hashable

Serialized parts of a feedback function.

The non-serialized parts are in the Feedback class.

Attributes¶

tru_class_info `instance-attribute` ¶

tru_class_info: Class

Class information of this pydantic object for use in deserialization.

Using this odd key to not pollute attribute names in whatever class we mix this into. Should be the same as CLASS_INFO.

implementation `class-attribute` `instance-attribute` ¶

implementation: Optional[Union[Function, Method]] = None

Implementation serialization.

aggregator `class-attribute` `instance-attribute` ¶

aggregator: Optional[Union[Function, Method]] = None

Aggregator method serialization.

combinations `class-attribute` `instance-attribute` ¶

combinations: Optional[FeedbackCombinations] = PRODUCT

Mode of combining selected values to produce arguments to each feedback function call.

feedback_definition_id `instance-attribute` ¶

feedback_definition_id: FeedbackDefinitionID = (
    feedback_definition_id
)

Id, if not given, uniquely determined from content.

if_exists `class-attribute` `instance-attribute` ¶

if_exists: Optional[Lens] = None

Only execute the feedback function if the following selector names something that exists in a record/app.

Can use this to evaluate conditionally on presence of some calls, for example. Feedbacks skipped this way will have a status of FeedbackResultStatus.SKIPPED.

if_missing `class-attribute` `instance-attribute` ¶

if_missing: FeedbackOnMissingParameters = ERROR

How to handle missing parameters in feedback function calls.

run_location `instance-attribute` ¶

run_location: Optional[FeedbackRunLocation]

Where the feedback evaluation takes place (e.g. locally, at a Snowflake server, etc).

selectors `instance-attribute` ¶

selectors: Dict[str, Lens]

Selectors; pointers into Records of where to get arguments for imp.

supplied_name `class-attribute` `instance-attribute` ¶

supplied_name: Optional[str] = None

An optional name. Only will affect displayed tables.

higher_is_better `class-attribute` `instance-attribute` ¶

higher_is_better: Optional[bool] = None

Feedback result magnitude interpretation.

name `property` ¶

name: str

Name of the feedback function.

Derived from the name of the serialized implementation function if name was not provided.

Functions¶

__rich_repr__ ¶

__rich_repr__() -> Result

Requirement for pretty printing using the rich package.

load `staticmethod` ¶

load(obj, *args, **kwargs)

Deserialize/load this object using the class information in tru_class_info to lookup the actual class that will do the deserialization.

model_validate `classmethod` ¶

model_validate(*args, **kwargs) -> Any

Deserialized a jsonized version of the app into the instance of the class it was serialized from.

Note

This process uses extra information stored in the jsonized object and handled by WithClassInfo.

FeedbackMode ¶

Bases: str, Enum

Mode of feedback evaluation.

Specify this using the feedback_mode to App constructors.

Note

This class extends str to allow users to compare its values with their string representations, i.e. in if mode == "none": .... Internal uses should use the enum instances.

Attributes¶

NONE `class-attribute` `instance-attribute` ¶

NONE = 'none'

No evaluation will happen even if feedback functions are specified.

WITH_APP `class-attribute` `instance-attribute` ¶

WITH_APP = 'with_app'

Try to run feedback functions immediately and before app returns a record.

WITH_APP_THREAD `class-attribute` `instance-attribute` ¶

WITH_APP_THREAD = 'with_app_thread'

Try to run feedback functions in the same process as the app but after it produces a record.

DEFERRED `class-attribute` `instance-attribute` ¶

DEFERRED = 'deferred'

Evaluate later via the process started by TruSession.start_deferred_feedback_evaluator.

FeedbackResult ¶

Bases: SerialModel

Feedback results for a single Feedback instance.

This might involve multiple feedback function calls. Typically you should not be constructing these objects yourself except for the cases where you'd like to log human feedback.

ATTRIBUTE	DESCRIPTION
`feedback_result_id`	Unique identifier for this result. TYPE: `str`
`record_id`	Record over which the feedback was evaluated. TYPE: `str`
`feedback_definition_id`	The id of the FeedbackDefinition which was evaluated to get this result. TYPE: `str`
`last_ts`	Last timestamp involved in the evaluation. TYPE: `datetime`
`status`	For deferred feedback evaluation, the status of the evaluation. TYPE: `FeedbackResultStatus`
`cost`	Cost of the evaluation. TYPE: `Cost`
`name`	Given name of the feedback. TYPE: `str`
`calls`	Individual feedback function invocations. TYPE: `List[FeedbackCall]`
`result`	Final result, potentially aggregating multiple calls. TYPE: `float`
`error`	Error information if there was an error. TYPE: `str`
`multi_result`	TODO: doc TYPE: `str`

Attributes¶

status `class-attribute` `instance-attribute` ¶

status: FeedbackResultStatus = NONE

For deferred feedback evaluation, the status of the evaluation.

Functions¶

__rich_repr__ ¶

__rich_repr__() -> Result

Requirement for pretty printing using the rich package.

Record ¶

Bases: SerialModel, Hashable

The record of a single main method call.

Note

This class will be renamed to Trace in the future.

Attributes¶

record_id `instance-attribute` ¶

record_id: RecordID = record_id

Unique identifier for this record.

app_id `instance-attribute` ¶

app_id: AppID

The app that produced this record.

cost `class-attribute` `instance-attribute` ¶

cost: Optional[Cost] = None

Costs associated with the record.

perf `class-attribute` `instance-attribute` ¶

perf: Optional[Perf] = None

Performance information.

ts `class-attribute` `instance-attribute` ¶

ts: datetime = Field(default_factory=now)

Timestamp of last update.

This is usually set whenever a record is changed in any way.

tags `class-attribute` `instance-attribute` ¶

tags: Optional[str] = ''

Tags for the record.

meta `class-attribute` `instance-attribute` ¶

meta: Optional[JSON] = None

Metadata for the record.

main_input `class-attribute` `instance-attribute` ¶

main_input: Optional[JSON] = None

The app's main input.

main_output `class-attribute` `instance-attribute` ¶

main_output: Optional[JSON] = None

The app's main output if there was no error.

main_error `class-attribute` `instance-attribute` ¶

main_error: Optional[JSON] = None

The app's main error if there was an error.

calls `class-attribute` `instance-attribute` ¶

calls: List[RecordAppCall] = []

The collection of calls recorded.

Note that these can be converted into a json structure with the same paths as the app that generated this record via layout_calls_as_app.

feedback_and_future_results `class-attribute` `instance-attribute` ¶

feedback_and_future_results: Optional[
    List[Tuple[FeedbackDefinition, Future[FeedbackResult]]]
] = Field(None, exclude=True)

Map of feedbacks to the futures for of their results.

These are only filled for records that were just produced. This will not be filled in when read from database. Also, will not fill in when using FeedbackMode.DEFERRED.

feedback_results `class-attribute` `instance-attribute` ¶

feedback_results: Optional[List[Future[FeedbackResult]]] = (
    Field(None, exclude=True)
)

Only the futures part of the above for backwards compatibility.

Functions¶

__rich_repr__ ¶

__rich_repr__() -> Result

Requirement for pretty printing using the rich package.

wait_for_feedback_results ¶

wait_for_feedback_results(
    feedback_timeout: Optional[float] = None,
) -> Dict[FeedbackDefinition, FeedbackResult]

Wait for feedback results to finish.

PARAMETER	DESCRIPTION
`feedback_timeout`	Timeout in seconds for each feedback function. If not given, will use the default timeout `trulens.core.utils.threading.TP.DEBUG_TIMEOUT`. TYPE: `Optional[float]` DEFAULT: `None`

RETURNS	DESCRIPTION
`Dict[FeedbackDefinition, FeedbackResult]`	A mapping of feedback functions to their results.

layout_calls_as_app ¶

layout_calls_as_app() -> Munch

Layout the calls in this record into the structure that follows that of the app that created this record.

This uses the paths stored in each RecordAppCall which are paths into the app.

Note: We cannot create a validated AppDefinition class (or subclass) object here as the layout of records differ in these ways:

Records do not include anything that is not an instrumented method hence have most of the structure of a app missing.
Records have RecordAppCall as their leafs where method definitions would be in the AppDefinition structure.

Select ¶

Utilities for creating selectors using Lens and aliases/shortcuts.

Attributes¶

Query `class-attribute` `instance-attribute` ¶

Query = Lens

Selector type.

Tru `class-attribute` `instance-attribute` ¶

Tru: Lens = Query()

Selector for the tru wrapper (TruLlama, TruChain, etc.).

Record `class-attribute` `instance-attribute` ¶

Record: Query = __record__

Selector for the record.

App `class-attribute` `instance-attribute` ¶

App: Query = __app__

Selector for the app.

RecordInput `class-attribute` `instance-attribute` ¶

RecordInput: Query = main_input

Selector for the main app input.

RecordOutput `class-attribute` `instance-attribute` ¶

RecordOutput: Query = main_output

Selector for the main app output.

RecordCalls `class-attribute` `instance-attribute` ¶

RecordCalls: Query = app

Selector for the calls made by the wrapped app.

Laid out by path into components.

RecordCall `class-attribute` `instance-attribute` ¶

RecordCall: Query = calls[-1]

Selector for the first called method (last to return).

RecordArgs `class-attribute` `instance-attribute` ¶

RecordArgs: Query = args

Selector for the whole set of inputs/arguments to the first called / last method call.

RecordRets `class-attribute` `instance-attribute` ¶

RecordRets: Query = rets

Selector for the whole output of the first called / last returned method call.

Functions¶

path_and_method `staticmethod` ¶

path_and_method(select: Query) -> Tuple[Query, str]

If select names in method as the last attribute, extract the method name and the selector without the final method name.

dequalify `staticmethod` ¶

dequalify(select: Query) -> Query

If the given selector qualifies record or app, remove that qualification.

render_for_dashboard `staticmethod` ¶

render_for_dashboard(query: Query) -> str

Render the given query for use in dashboard to help user specify feedback functions.

trulens.core.schema¶