trulens.core.schema.feedback¶

trulens.core.schema.feedback ¶

Serializable feedback-related classes.

Classes¶

FeedbackMode ¶

Bases: str, Enum

Mode of feedback evaluation.

Specify this using the feedback_mode to App constructors.

Note

This class extends str to allow users to compare its values with their string representations, i.e. in if mode == "none": .... Internal uses should use the enum instances.

Attributes¶

NONE `class-attribute` `instance-attribute` ¶

NONE = 'none'

No evaluation will happen even if feedback functions are specified.

WITH_APP `class-attribute` `instance-attribute` ¶

WITH_APP = 'with_app'

Try to run feedback functions immediately and before app returns a record.

WITH_APP_THREAD `class-attribute` `instance-attribute` ¶

WITH_APP_THREAD = 'with_app_thread'

Try to run feedback functions in the same process as the app but after it produces a record.

DEFERRED `class-attribute` `instance-attribute` ¶

DEFERRED = 'deferred'

Evaluate later via the process started by TruSession.start_deferred_feedback_evaluator.

FeedbackRunLocation ¶

Bases: str, Enum

Where the feedback evaluation takes place (e.g. locally, at a Snowflake server, etc).

Attributes¶

IN_APP `class-attribute` `instance-attribute` ¶

IN_APP = 'in_app'

Run on the same process (or child process) of the app invocation.

SNOWFLAKE `class-attribute` `instance-attribute` ¶

SNOWFLAKE = 'snowflake'

Run on a Snowflake server.

FeedbackResultStatus ¶

Bases: str, Enum

For deferred feedback evaluation, these values indicate status of evaluation.

Note

This class extends str to allow users to compare its values with their string representations, i.e. in if status == "done": .... Internal uses should use the enum instances.

Attributes¶

NONE `class-attribute` `instance-attribute` ¶

NONE = 'none'

Initial value is none.

RUNNING `class-attribute` `instance-attribute` ¶

RUNNING = 'running'

Once queued/started, status is updated to "running".

FAILED `class-attribute` `instance-attribute` ¶

FAILED = 'failed'

Run failed.

DONE `class-attribute` `instance-attribute` ¶

DONE = 'done'

Run completed successfully.

SKIPPED `class-attribute` `instance-attribute` ¶

SKIPPED = 'skipped'

This feedback was skipped.

This can be because because it had an if_exists selector and did not select anything or it has a selector that did not select anything the on_missing was set to warn or ignore.

FeedbackOnMissingParameters ¶

Bases: str, Enum

How to handle missing parameters in feedback function calls.

This is specifically for the case were a feedback function has a selector that selects something that does not exist in a record/app.

Note

This class extends str to allow users to compare its values with their string representations, i.e. in if onmissing == "error": .... Internal uses should use the enum instances.

Attributes¶

ERROR `class-attribute` `instance-attribute` ¶

ERROR = 'error'

Raise an error if a parameter is missing.

The result status will be set to FAILED.

WARN `class-attribute` `instance-attribute` ¶

WARN = 'warn'

Warn if a parameter is missing.

The result status will be set to SKIPPED.

IGNORE `class-attribute` `instance-attribute` ¶

IGNORE = 'ignore'

Do nothing.

No warning or error message will be shown. The result status will be set to SKIPPED.

FeedbackCall ¶

Bases: SerialModel

Invocations of feedback function results in one of these instances.

Note that a single Feedback instance might require more than one call.

Attributes¶

args `instance-attribute` ¶

args: Dict[str, Optional[JSON]]

Arguments to the feedback function.

ret `instance-attribute` ¶

ret: Union[float, List[float], List[Tuple]]

Return value.

meta `class-attribute` `instance-attribute` ¶

meta: Dict[str, Any] = Field(default_factory=dict)

Any additional data a feedback function returns to display alongside its float result.

Functions¶

__rich_repr__ ¶

__rich_repr__() -> Result

Requirement for pretty printing using the rich package.

FeedbackResult ¶

Bases: SerialModel

Feedback results for a single Feedback instance.

This might involve multiple feedback function calls. Typically you should not be constructing these objects yourself except for the cases where you'd like to log human feedback.

ATTRIBUTE	DESCRIPTION
`feedback_result_id`	Unique identifier for this result. TYPE: `str`
`record_id`	Record over which the feedback was evaluated. TYPE: `str`
`feedback_definition_id`	The id of the FeedbackDefinition which was evaluated to get this result. TYPE: `str`
`last_ts`	Last timestamp involved in the evaluation. TYPE: `datetime`
`status`	For deferred feedback evaluation, the status of the evaluation. TYPE: `FeedbackResultStatus`
`cost`	Cost of the evaluation. TYPE: `Cost`
`name`	Given name of the feedback. TYPE: `str`
`calls`	Individual feedback function invocations. TYPE: `List[FeedbackCall]`
`result`	Final result, potentially aggregating multiple calls. TYPE: `float`
`error`	Error information if there was an error. TYPE: `str`
`multi_result`	TODO: doc TYPE: `str`

Attributes¶

status `class-attribute` `instance-attribute` ¶

status: FeedbackResultStatus = NONE

For deferred feedback evaluation, the status of the evaluation.

Functions¶

__rich_repr__ ¶

__rich_repr__() -> Result

Requirement for pretty printing using the rich package.

FeedbackCombinations ¶

Bases: str, Enum

How to collect arguments for feedback function calls.

Note that this applies only to cases where selectors pick out more than one thing for feedback function arguments. This option is used for the field combinations of FeedbackDefinition and can be specified with Feedback.aggregate.

Attributes¶

ZIP `class-attribute` `instance-attribute` ¶

ZIP = 'zip'

Match argument values per position in produced values.

Example

If the selector for arg1 generates values 0, 1, 2 and one for arg2 generates values "a", "b", "c", the feedback function will be called 3 times with kwargs:

{'arg1': 0, arg2: "a"},
{'arg1': 1, arg2: "b"},
{'arg1': 2, arg2: "c"}

If the quantities of items in the various generators do not match, the result will have only as many combinations as the generator with the fewest items as per python zip (strict mode is not used).

Note that selectors can use Lens collect() to name a single (list) value instead of multiple values.

PRODUCT `class-attribute` `instance-attribute` ¶

PRODUCT = 'product'

Evaluate feedback on all combinations of feedback function arguments.

Example

If the selector for arg1 generates values 0, 1 and the one for arg2 generates values "a", "b", the feedback function will be called 4 times with kwargs:

{'arg1': 0, arg2: "a"},
{'arg1': 0, arg2: "b"},
{'arg1': 1, arg2: "a"},
{'arg1': 1, arg2: "b"}

See itertools.product for more.

Note that selectors can use Lens collect() to name a single (list) value instead of multiple values.

FeedbackDefinition ¶

Bases: WithClassInfo, SerialModel, Hashable

Serialized parts of a feedback function.

The non-serialized parts are in the Feedback class.

Attributes¶

tru_class_info `instance-attribute` ¶

tru_class_info: Class

Class information of this pydantic object for use in deserialization.

Using this odd key to not pollute attribute names in whatever class we mix this into. Should be the same as CLASS_INFO.

implementation `class-attribute` `instance-attribute` ¶

implementation: Optional[Union[Function, Method]] = None

Implementation serialization.

aggregator `class-attribute` `instance-attribute` ¶

aggregator: Optional[Union[Function, Method]] = None

Aggregator method serialization.

combinations `class-attribute` `instance-attribute` ¶

combinations: Optional[FeedbackCombinations] = PRODUCT

Mode of combining selected values to produce arguments to each feedback function call.

if_exists `class-attribute` `instance-attribute` ¶

if_exists: Optional[Lens] = None

Only execute the feedback function if the following selector names something that exists in a record/app.

Can use this to evaluate conditionally on presence of some calls, for example. Feedbacks skipped this way will have a status of FeedbackResultStatus.SKIPPED.

if_missing `class-attribute` `instance-attribute` ¶

if_missing: FeedbackOnMissingParameters = ERROR

How to handle missing parameters in feedback function calls.

run_location `instance-attribute` ¶

run_location: Optional[FeedbackRunLocation]

Where the feedback evaluation takes place (e.g. locally, at a Snowflake server, etc).

selectors `instance-attribute` ¶

selectors: Dict[str, Lens]

Selectors; pointers into Records of where to get arguments for imp.

supplied_name `class-attribute` `instance-attribute` ¶

supplied_name: Optional[str] = None

An optional name. Only will affect displayed tables.

higher_is_better `class-attribute` `instance-attribute` ¶

higher_is_better: Optional[bool] = None

Feedback result magnitude interpretation.

feedback_definition_id `instance-attribute` ¶

feedback_definition_id: FeedbackDefinitionID = (
    feedback_definition_id
)

Id, if not given, uniquely determined from content.

name `property` ¶

name: str

Name of the feedback function.

Derived from the name of the serialized implementation function if name was not provided.

Functions¶

__rich_repr__ ¶

__rich_repr__() -> Result

Requirement for pretty printing using the rich package.

load `staticmethod` ¶

load(obj, *args, **kwargs)

Deserialize/load this object using the class information in tru_class_info to lookup the actual class that will do the deserialization.

model_validate `classmethod` ¶

model_validate(*args, **kwargs) -> Any

Deserialized a jsonized version of the app into the instance of the class it was serialized from.

Note

This process uses extra information stored in the jsonized object and handled by WithClassInfo.

trulens.core.schema.feedback¶