OpenAI Assistants API¶

The Assistants API allows you to build AI assistants within your own applications. An Assistant has instructions and can leverage models, tools, and knowledge to respond to user queries. The Assistants API currently supports three types of tools: Code Interpreter, Retrieval, and Function calling.

TruLens can be easily integrated with the assistants API to provide the same observability tooling you are used to when building with other frameworks.

[Important] Notice in this example notebook, we are using Assistants API V1 (hence the pinned version of openai below) so that we can evaluate against retrieved source. At some very recent point in time as of April 2024, OpenAI removed the "quote" attribute from file citation object in Assistants API V2 due to stability issue of this feature. See response from OpenAI staff https://community.openai.com/t/assistant-api-always-return-empty-annotations/489285/48

Here's the migration guide for easier navigating between V1 and V2 of Assistants API: https://platform.openai.com/docs/assistants/migration/changing-beta-versions

In [ ]:

# !pip install trulens trulens-providers-openai openai==1.14.3 # pinned openai version to avoid breaking changes

# !pip install trulens trulens-providers-openai openai==1.14.3 # pinned openai version to avoid breaking changes

Set keys¶

In [ ]:

import os

os.environ["OPENAI_API_KEY"] = "sk-..."

import os os.environ["OPENAI_API_KEY"] = "sk-..."

Create the assistant¶

Let's create a new assistant that answers questions about the famous Paul Graham Essay.

The easiest way to get it is to download it via this link and save it in a folder called data. You can do so with the following command

In [ ]:

!wget https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt -P data/

!wget https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt -P data/

Add TruLens¶

In [ ]:

from trulens.core import TruSession
from trulens.apps.custom import instrument

session = TruSession()
session.reset_database()

from trulens.core import TruSession from trulens.apps.custom import instrument session = TruSession() session.reset_database()

Create a thread (V1 Assistants)¶

In [ ]:

from openai import OpenAI


class RAG_with_OpenAI_Assistant:
    def __init__(self):
        client = OpenAI()
        self.client = client

        # upload the file\
        file = client.files.create(
            file=open("data/paul_graham_essay.txt", "rb"), purpose="assistants"
        )

        # create the assistant with access to a retrieval tool
        assistant = client.beta.assistants.create(
            name="Paul Graham Essay Assistant",
            instructions="You are an assistant that answers questions about Paul Graham.",
            tools=[{"type": "retrieval"}],
            model="gpt-4-turbo-preview",
            file_ids=[file.id],
        )

        self.assistant = assistant

    @instrument
    def retrieve_and_generate(self, query: str) -> str:
        """
        Retrieve relevant text by creating and running a thread with the OpenAI assistant.
        """
        self.thread = self.client.beta.threads.create()
        self.message = self.client.beta.threads.messages.create(
            thread_id=self.thread.id, role="user", content=query
        )

        run = self.client.beta.threads.runs.create(
            thread_id=self.thread.id,
            assistant_id=self.assistant.id,
            instructions="Please answer any questions about Paul Graham.",
        )

        # Wait for the run to complete
        import time

        while run.status in ["queued", "in_progress", "cancelling"]:
            time.sleep(1)
            run = self.client.beta.threads.runs.retrieve(
                thread_id=self.thread.id, run_id=run.id
            )

        if run.status == "completed":
            messages = self.client.beta.threads.messages.list(
                thread_id=self.thread.id
            )
            response = messages.data[0].content[0].text.value
            quote = (
                messages.data[0]
                .content[0]
                .text.annotations[0]
                .file_citation.quote
            )
        else:
            response = "Unable to retrieve information at this time."

        return response, quote


rag = RAG_with_OpenAI_Assistant()

from openai import OpenAI class RAG_with_OpenAI_Assistant: def __init__(self): client = OpenAI() self.client = client # upload the file\ file = client.files.create( file=open("data/paul_graham_essay.txt", "rb"), purpose="assistants" ) # create the assistant with access to a retrieval tool assistant = client.beta.assistants.create( name="Paul Graham Essay Assistant", instructions="You are an assistant that answers questions about Paul Graham.", tools=[{"type": "retrieval"}], model="gpt-4-turbo-preview", file_ids=[file.id], ) self.assistant = assistant @instrument def retrieve_and_generate(self, query: str) -> str: """ Retrieve relevant text by creating and running a thread with the OpenAI assistant. """ self.thread = self.client.beta.threads.create() self.message = self.client.beta.threads.messages.create( thread_id=self.thread.id, role="user", content=query ) run = self.client.beta.threads.runs.create( thread_id=self.thread.id, assistant_id=self.assistant.id, instructions="Please answer any questions about Paul Graham.", ) # Wait for the run to complete import time while run.status in ["queued", "in_progress", "cancelling"]: time.sleep(1) run = self.client.beta.threads.runs.retrieve( thread_id=self.thread.id, run_id=run.id ) if run.status == "completed": messages = self.client.beta.threads.messages.list( thread_id=self.thread.id ) response = messages.data[0].content[0].text.value quote = ( messages.data[0] .content[0] .text.annotations[0] .file_citation.quote ) else: response = "Unable to retrieve information at this time." return response, quote rag = RAG_with_OpenAI_Assistant()

Create feedback functions¶

In [ ]:

import numpy as np
from trulens.core import Feedback
from trulens.core import Select
from trulens.providers.openai import OpenAI as fOpenAI

provider = fOpenAI()


# Define a groundedness feedback function
f_groundedness = (
    Feedback(
        provider.groundedness_measure_with_cot_reasons, name="Groundedness"
    )
    .on(Select.RecordCalls.retrieve_and_generate.rets[1])
    .on(Select.RecordCalls.retrieve_and_generate.rets[0])
)

# Question/answer relevance between overall question and answer.
f_answer_relevance = (
    Feedback(provider.relevance_with_cot_reasons, name="Answer Relevance")
    .on(Select.RecordCalls.retrieve_and_generate.args.query)
    .on(Select.RecordCalls.retrieve_and_generate.rets[0])
)

# Question/statement relevance between question and each context chunk.
f_context_relevance = (
    Feedback(
        provider.context_relevance_with_cot_reasons, name="Context Relevance"
    )
    .on(Select.RecordCalls.retrieve_and_generate.args.query)
    .on(Select.RecordCalls.retrieve_and_generate.rets[1])
    .aggregate(np.mean)
)

import numpy as np from trulens.core import Feedback from trulens.core import Select from trulens.providers.openai import OpenAI as fOpenAI provider = fOpenAI() # Define a groundedness feedback function f_groundedness = ( Feedback( provider.groundedness_measure_with_cot_reasons, name="Groundedness" ) .on(Select.RecordCalls.retrieve_and_generate.rets[1]) .on(Select.RecordCalls.retrieve_and_generate.rets[0]) ) # Question/answer relevance between overall question and answer. f_answer_relevance = ( Feedback(provider.relevance_with_cot_reasons, name="Answer Relevance") .on(Select.RecordCalls.retrieve_and_generate.args.query) .on(Select.RecordCalls.retrieve_and_generate.rets[0]) ) # Question/statement relevance between question and each context chunk. f_context_relevance = ( Feedback( provider.context_relevance_with_cot_reasons, name="Context Relevance" ) .on(Select.RecordCalls.retrieve_and_generate.args.query) .on(Select.RecordCalls.retrieve_and_generate.rets[1]) .aggregate(np.mean) )

In [ ]:

from trulens.apps.custom import TruCustomApp

tru_rag = TruCustomApp(
    rag,
    app_name="OpenAI Assistant RAG",
    feedbacks=[f_groundedness, f_answer_relevance, f_context_relevance],
)

from trulens.apps.custom import TruCustomApp tru_rag = TruCustomApp( rag, app_name="OpenAI Assistant RAG", feedbacks=[f_groundedness, f_answer_relevance, f_context_relevance], )

In [ ]:

with tru_rag:
    rag.retrieve_and_generate("How did paul graham grow up?")

with tru_rag: rag.retrieve_and_generate("How did paul graham grow up?")

In [ ]:

session.get_leaderboard()

session.get_leaderboard()

In [ ]:

from trulens.dashboard import run_dashboard

run_dashboard()

from trulens.dashboard import run_dashboard run_dashboard()