Wandb回调处理程序¶
Weights & Biases Prompts 是一套专为开发LLM驱动应用程序而构建的LLMOps工具套件。
WandbCallbackHandler
与W&B Prompts集成,用于可视化和检查索引构建的执行流程,或者在索引上进行查询等。您可以使用此处理程序将创建的索引持久化为W&B Artifacts,从而实现对索引的版本控制。
In [ ]:
Copied!
%pip install llama-index-callbacks-wandb
%pip install llama-index-llms-openai
%pip install llama-index-callbacks-wandb
%pip install llama-index-llms-openai
In [ ]:
Copied!
import os
from getpass import getpass
if os.getenv("OPENAI_API_KEY") is None:
os.environ["OPENAI_API_KEY"] = getpass(
"Paste your OpenAI key from:"
" https://platform.openai.com/account/api-keys\n"
)
assert os.getenv("OPENAI_API_KEY", "").startswith(
"sk-"
), "This doesn't look like a valid OpenAI API key"
print("OpenAI API key configured")
import os
from getpass import getpass
if os.getenv("OPENAI_API_KEY") is None:
os.environ["OPENAI_API_KEY"] = getpass(
"Paste your OpenAI key from:"
" https://platform.openai.com/account/api-keys\n"
)
assert os.getenv("OPENAI_API_KEY", "").startswith(
"sk-"
), "This doesn't look like a valid OpenAI API key"
print("OpenAI API key configured")
OpenAI API key configured
In [ ]:
Copied!
from llama_index.core.callbacks import CallbackManager
from llama_index.core.callbacks import LlamaDebugHandler
from llama_index.callbacks.wandb import WandbCallbackHandler
from llama_index.core import (
VectorStoreIndex,
SimpleDirectoryReader,
SimpleKeywordTableIndex,
StorageContext,
)
from llama_index.llms.openai import OpenAI
from llama_index.core.callbacks import CallbackManager
from llama_index.core.callbacks import LlamaDebugHandler
from llama_index.callbacks.wandb import WandbCallbackHandler
from llama_index.core import (
VectorStoreIndex,
SimpleDirectoryReader,
SimpleKeywordTableIndex,
StorageContext,
)
from llama_index.llms.openai import OpenAI
设置LLM¶
In [ ]:
Copied!
from llama_index.core import Settings
Settings.llm = OpenAI(model="gpt-4", temperature=0)
from llama_index.core import Settings
Settings.llm = OpenAI(model="gpt-4", temperature=0)
W&B Callback Manager设置¶
选项1:设置全局评估处理程序
In [ ]:
Copied!
import llama_index.core
from llama_index.core import set_global_handler
set_global_handler("wandb", run_args={"project": "llamaindex"})
wandb_callback = llama_index.core.global_handler
import llama_index.core
from llama_index.core import set_global_handler
set_global_handler("wandb", run_args={"project": "llamaindex"})
wandb_callback = llama_index.core.global_handler
选项 2:手动配置回调处理程序
还可以为额外的笔记本可见性配置调试器处理程序。
In [ ]:
Copied!
llama_debug = LlamaDebugHandler(print_trace_on_end=True)
# wandb.init 参数
run_args = dict(
project="llamaindex",
)
wandb_callback = WandbCallbackHandler(run_args=run_args)
Settings.callback_manager = CallbackManager([llama_debug, wandb_callback])
llama_debug = LlamaDebugHandler(print_trace_on_end=True)
# wandb.init 参数
run_args = dict(
project="llamaindex",
)
wandb_callback = WandbCallbackHandler(run_args=run_args)
Settings.callback_manager = CallbackManager([llama_debug, wandb_callback])
运行上面的单元格后,您将获得W&B运行页面的URL。在这里,您将找到一个跟踪表,其中列出了使用Weights and Biases的Prompts功能跟踪的所有事件。
1. 索引¶
下载数据
In [ ]:
Copied!
!mkdir -p 'data/paul_graham/'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'
!mkdir -p 'data/paul_graham/'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'
In [ ]:
Copied!
docs = SimpleDirectoryReader("./data/paul_graham/").load_data()
docs = SimpleDirectoryReader("./data/paul_graham/").load_data()
In [ ]:
Copied!
index = VectorStoreIndex.from_documents(docs)
index = VectorStoreIndex.from_documents(docs)
********** Trace: index_construction |_node_parsing -> 0.295179 seconds |_chunking -> 0.293976 seconds |_embedding -> 0.494492 seconds |_embedding -> 0.346162 seconds **********
wandb: Logged trace tree to W&B.
1.1 将索引持久化为W&B工件¶
In [ ]:
Copied!
wandb_callback.persist_index(index, index_name="simple_vector_store")
wandb_callback.persist_index(index, index_name="simple_vector_store")
wandb: Adding directory to artifact (/Users/loganmarkewich/llama_index/docs/examples/callbacks/wandb/run-20230801_152955-ds93prxa/files/storage)... Done. 0.0s
1.2 从W&B Artifacts下载索引¶
In [ ]:
Copied!
from llama_index.core import load_index_from_storage
storage_context = wandb_callback.load_storage_context(
artifact_url="ayut/llamaindex/simple_vector_store:v0"
)
# 加载索引并初始化查询引擎
index = load_index_from_storage(
storage_context,
)
from llama_index.core import load_index_from_storage
storage_context = wandb_callback.load_storage_context(
artifact_url="ayut/llamaindex/simple_vector_store:v0"
)
# 加载索引并初始化查询引擎
index = load_index_from_storage(
storage_context,
)
wandb: 3 of 3 files downloaded.
********** Trace: index_construction **********
2. 在索引上进行查询¶
In [ ]:
Copied!
query_engine = index.as_query_engine()
response = query_engine.query("What did the author do growing up?")
print(response, sep="\n")
query_engine = index.as_query_engine()
response = query_engine.query("What did the author do growing up?")
print(response, sep="\n")
********** Trace: query |_query -> 2.695958 seconds |_retrieve -> 0.806379 seconds |_embedding -> 0.802871 seconds |_synthesize -> 1.8893 seconds |_llm -> 1.842434 seconds **********
wandb: Logged trace tree to W&B.
The text does not provide information on what the author did growing up.
关闭 W&B 回调处理程序¶
当我们完成跟踪事件时,可以关闭 wandb 运行。
In [ ]:
Copied!
wandb_callback.finish()
wandb_callback.finish()