跳到主要内容

使用LangChain、Deep Lake和OpenAI进行问答

nbviewer

这个笔记本展示了如何使用LangChain、Deep Lake作为向量存储和OpenAI嵌入来实现一个问答系统。我们将采取以下步骤来实现这一目标：

  1. 加载一个Deep Lake文本数据集
  2. 使用LangChain初始化Deep Lake向量存储
  3. 将文本添加到向量存储中
  4. 在数据库上运行查询
  5. 完成！

您也可以参考其他教程，比如在Deep Lake中存储的任何类型数据（PDF、json、csv、文本）上进行问答：与任何数据交谈，代码理解，或者在PDF上进行问答，或者推荐歌曲。

安装要求​

让我们安装以下包。

!pip install deeplake langchain openai tiktoken

认证​

在这里提供你的OpenAI API密钥：

import getpass
import os

os.environ['OPENAI_API_KEY'] = getpass.getpass()

··········

加载一个深湖文本数据集​

我们将在这个示例中使用cohere-wikipedia-22数据集的20000个样本子集。

import deeplake

ds = deeplake.load("hub://activeloop/cohere-wikipedia-22-sample")
ds.summary()

\
Opening dataset in read-only mode as you don't have write permissions.
-
This dataset can be visualized in Jupyter Notebook by ds.visualize() or at https://app.activeloop.ai/activeloop/cohere-wikipedia-22-sample
|
hub://activeloop/cohere-wikipedia-22-sample loaded successfully.

Dataset(path='hub://activeloop/cohere-wikipedia-22-sample', read_only=True, tensors=['ids', 'metadata', 'text'])

  tensor    htype     shape      dtype  compression
 -------   -------   -------    -------  ------- 
   ids      text    (20000, 1)    str     None   
 metadata   json    (20000, 1)    str     None   
   text     text    (20000, 1)    str     None

让我们来看几个示例：

ds[:3].text.data()["value"]

['The 24-hour clock is a way of telling the time in which the day runs from midnight to midnight and is divided into 24 hours, numbered from 0 to 23. It does not use a.m. or p.m. This system is also referred to (only in the US and the English speaking parts of Canada) as military time or (only in the United Kingdom and now very rarely) as continental time. In some parts of the world, it is called railway time. Also, the international standard notation of time (ISO 8601) is based on this format.',
 'A time in the 24-hour clock is written in the form hours:minutes (for example, 01:23), or hours:minutes:seconds (01:23:45). Numbers under 10 have a zero in front (called a leading zero); e.g. 09:07. Under the 24-hour clock system, the day begins at midnight, 00:00, and the last minute of the day begins at 23:59 and ends at 24:00, which is identical to 00:00 of the following day. 12:00 can only be mid-day. Midnight is called 24:00 and is used to mean the end of the day and 00:00 is used to mean the beginning of the day. For example, you would say "Tuesday at 24:00" and "Wednesday at 00:00" to mean exactly the same time.',
 'However, the US military prefers not to say 24:00 - they do not like to have two names for the same thing, so they always say "23:59", which is one minute before midnight.']

LangChain的Deep Lake向量存储​

让我们定义一个dataset_path，这是您的Deep Lake向量存储将存放文本嵌入的位置。

dataset_path = 'wikipedia-embeddings-deeplake'

我们将设置OpenAI的text-embedding-3-small作为我们的嵌入函数，并在dataset_path上初始化一个Deep Lake向量存储…

from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores import DeepLake

embedding = OpenAIEmbeddings(model="text-embedding-3-small")
db = DeepLake(dataset_path, embedding=embedding, overwrite=True)

…并使用add_texts方法逐批次填充样本。

from tqdm.auto import tqdm

batch_size = 100

nsamples = 10  # 用于测试。将此替换为 len(ds) 以追加所有内容。
for i in tqdm(range(0, nsamples, batch_size)):
    # 查找批处理结束位置
    i_end = min(nsamples, i + batch_size)

    batch = ds[i:i_end]
    id_batch = batch.ids.data()["value"]
    text_batch = batch.text.data()["value"]
    meta_batch = batch.metadata.data()["value"]

    db.add_texts(text_batch, metadatas=meta_batch, ids=id_batch)

  0%|          | 0/1 [00:00<?, ?it/s]

creating embeddings:   0%|          | 0/1 [00:00<?, ?it/s]
creating embeddings: 100%|██████████| 1/1 [00:02<00:00,  2.11s/it]

100%|██████████| 10/10 [00:00<00:00, 462.42it/s]
Dataset(path='wikipedia-embeddings-deeplake', tensors=['text', 'metadata', 'embedding', 'id'])

  tensor      htype      shape      dtype  compression
  -------    -------    -------    -------  ------- 
   text       text      (10, 1)      str     None   
 metadata     json      (10, 1)      str     None   
 embedding  embedding  (10, 1536)  float32   None   
    id        text      (10, 1)      str     None

在数据库上运行用户查询​

通过db.vectorstore.dataset可以访问底层的Deep Lake数据集对象，可以使用db.vectorstore.summary()来总结数据结构，该函数显示了4个张量和10个样本：

db.vectorstore.summary()

Dataset(path='wikipedia-embeddings-deeplake', tensors=['text', 'metadata', 'embedding', 'id'])

  tensor      htype      shape      dtype  compression
  -------    -------    -------    -------  ------- 
   text       text      (10, 1)      str     None   
 metadata     json      (10, 1)      str     None   
 embedding  embedding  (10, 1536)  float32   None   
    id        text      (10, 1)      str     None

我们现在将在我们的向量存储库中使用GPT-3.5-Turbo来设置问答系统。

from langchain.chains import RetrievalQA
from langchain.chat_models import ChatOpenAI

# Re-load the vector store in case it's no longer initialized
# db = DeepLake(dataset_path = dataset_path, embedding_function=embedding)

qa = RetrievalQA.from_chain_type(llm=ChatOpenAI(model='gpt-3.5-turbo'), chain_type="stuff", retriever=db.as_retriever())

让我们尝试运行一个提示并检查输出。在内部，该API执行嵌入式搜索，以找到最相关的数据并输入到LLM上下文中。

query = 'Why does the military not say 24:00?'
qa.run(query)

'The military prefers not to say 24:00 because they do not like to have two names for the same thing. Instead, they always say "23:59", which is one minute before midnight.'

Et voilà!