Redis向量存储¶
在这个笔记本中,我们将展示如何快速使用RedisVectorStore进行演示。
如果您在Colab上打开这个笔记本,您可能需要安装LlamaIndex 🦙。
%pip install -U llama-index llama-index-vector-stores-redis llama-index-embeddings-cohere llama-index-embeddings-openai
import osimport getpassimport sysimport loggingimport textwrapimport warningswarnings.filterwarnings("ignore")# 取消注释以查看调试日志logging.basicConfig(stream=sys.stdout, level=logging.INFO)from llama_index.core import VectorStoreIndex, SimpleDirectoryReaderfrom llama_index.vector_stores.redis import RedisVectorStore
启动Redis¶
启动Redis最简单的方法是使用Redis Stack docker镜像,或者快速注册一个免费的Redis Cloud实例。
要按照本教程的每一步操作,请按照以下方式启动镜像:
docker run --name redis-vecdb -d -p 6379:6379 -p 8001:8001 redis/redis-stack:latest
这也会在8001端口上启动RedisInsight UI,您可以在http://localhost:8001上查看。
设置OpenAI¶
首先,让我们添加OpenAI的API密钥。这将允许我们访问OpenAI以获取嵌入和使用ChatGPT。
oai_api_key = getpass.getpass("OpenAI API Key:")
os.environ["OPENAI_API_KEY"] = oai_api_key
下载数据
!mkdir -p 'data/paul_graham/'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'
--2024-04-10 19:35:33-- https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 2606:50c0:8003::154, 2606:50c0:8000::154, 2606:50c0:8002::154, ... Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|2606:50c0:8003::154|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 75042 (73K) [text/plain] Saving to: ‘data/paul_graham/paul_graham_essay.txt’ data/paul_graham/pa 100%[===================>] 73.28K --.-KB/s in 0.03s 2024-04-10 19:35:33 (2.15 MB/s) - ‘data/paul_graham/paul_graham_essay.txt’ saved [75042/75042]
读取数据集¶
在这里,我们将使用一组Paul Graham的文章作为文本数据集,将其转换为嵌入向量并存储在RedisVectorStore
中,然后查询以找到LLM QnA循环的上下文。
# 加载文档documents = SimpleDirectoryReader("./data/paul_graham").load_data()print( "文档ID:", documents[0].id_, "文档文件名:", documents[0].metadata["file_name"],)
Document ID: 7056f7ba-3513-4ef4-9792-2bd28040aaed Document Filename: paul_graham_essay.txt
初始化默认的Redis向量存储¶
现在我们已经准备好我们的文档,我们可以使用默认设置来初始化Redis向量存储。这将允许我们将向量存储在Redis中,并创建一个用于实时搜索的索引。
from llama_index.core import StorageContextfrom redis import Redis# 创建一个Redis客户端连接redis_client = Redis.from_url("redis://localhost:6379")# 创建向量存储包装器vector_store = RedisVectorStore(redis_client=redis_client, overwrite=True)# 加载存储上下文storage_context = StorageContext.from_defaults(vector_store=vector_store)# 从文档和存储上下文构建并加载索引index = VectorStoreIndex.from_documents( documents, storage_context=storage_context)# index = VectorStoreIndex.from_vector_store(vector_store=vector_store)
19:39:17 llama_index.vector_stores.redis.base INFO Using default RedisVectorStore schema. 19:39:19 httpx INFO HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" 19:39:19 llama_index.vector_stores.redis.base INFO Added 22 documents to index llama_index
查询默认的向量存储¶
现在我们已经将数据存储在索引中,我们可以对索引进行查询。
索引将使用数据作为LLM的知识库。as_query_engine()的默认设置利用OpenAI嵌入和GPT作为语言模型。因此,除非您选择自定义或本地语言模型,否则需要一个OpenAI密钥。
接下来,我们将对我们的索引进行搜索测试,然后使用LLM对整个RAG进行搜索。
query_engine = index.as_query_engine()
retriever = index.as_retriever()
result_nodes = retriever.retrieve("What did the author learn?")
for node in result_nodes:
print(node)
19:39:22 httpx INFO HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" 19:39:22 llama_index.vector_stores.redis.base INFO Querying index llama_index with filters * 19:39:22 llama_index.vector_stores.redis.base INFO Found 2 results for query with id ['llama_index/vector_adb6b7ce-49bb-4961-8506-37082c02a389', 'llama_index/vector_e39be1fe-32d0-456e-b211-4efabd191108'] Node ID: adb6b7ce-49bb-4961-8506-37082c02a389 Text: What I Worked On February 2021 Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I ... Score: 0.820 Node ID: e39be1fe-32d0-456e-b211-4efabd191108 Text: Except for a few officially anointed thinkers who went to the right parties in New York, the only people allowed to publish essays were specialists writing about their specialties. There were so many essays that had never been written, because there had been no way to publish them. Now they could be, and I was going to write them. [12] I've wor... Score: 0.819
response = query_engine.query("What did the author learn?")
print(textwrap.fill(str(response), 100))
19:39:25 httpx INFO HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" 19:39:25 llama_index.vector_stores.redis.base INFO Querying index llama_index with filters * 19:39:25 llama_index.vector_stores.redis.base INFO Found 2 results for query with id ['llama_index/vector_adb6b7ce-49bb-4961-8506-37082c02a389', 'llama_index/vector_e39be1fe-32d0-456e-b211-4efabd191108'] 19:39:27 httpx INFO HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" The author learned that working on things that weren't prestigious often led to valuable discoveries and indicated the right kind of motives. Despite the lack of initial prestige, pursuing such work could be a sign of genuine potential and appropriate motivations, steering clear of the common pitfall of being driven solely by the desire to impress others.
result_nodes = retriever.retrieve("What was a hard moment for the author?")
for node in result_nodes:
print(node)
19:39:27 httpx INFO HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" 19:39:27 llama_index.vector_stores.redis.base INFO Querying index llama_index with filters * 19:39:27 llama_index.vector_stores.redis.base INFO Found 2 results for query with id ['llama_index/vector_adb6b7ce-49bb-4961-8506-37082c02a389', 'llama_index/vector_e39be1fe-32d0-456e-b211-4efabd191108'] Node ID: adb6b7ce-49bb-4961-8506-37082c02a389 Text: What I Worked On February 2021 Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I ... Score: 0.802 Node ID: e39be1fe-32d0-456e-b211-4efabd191108 Text: Except for a few officially anointed thinkers who went to the right parties in New York, the only people allowed to publish essays were specialists writing about their specialties. There were so many essays that had never been written, because there had been no way to publish them. Now they could be, and I was going to write them. [12] I've wor... Score: 0.799
response = query_engine.query("What was a hard moment for the author?")
print(textwrap.fill(str(response), 100))
19:39:29 httpx INFO HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" 19:39:29 llama_index.vector_stores.redis.base INFO Querying index llama_index with filters * 19:39:29 llama_index.vector_stores.redis.base INFO Found 2 results for query with id ['llama_index/vector_adb6b7ce-49bb-4961-8506-37082c02a389', 'llama_index/vector_e39be1fe-32d0-456e-b211-4efabd191108'] 19:39:31 httpx INFO HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK" A hard moment for the author was when one of his programs on the IBM 1401 mainframe didn't terminate, leading to a technical error and an uncomfortable situation with the data center manager.
index.vector_store.delete_index()
19:39:34 llama_index.vector_stores.redis.base INFO Deleting index llama_index
from llama_index.core.settings import Settingsfrom llama_index.embeddings.cohere import CohereEmbedding# 设置Cohere密钥co_api_key = getpass.getpass("Cohere API Key:")os.environ["CO_API_KEY"] = co_api_key# 设置llamaindex使用Cohere嵌入Settings.embed_model = CohereEmbedding()
from redisvl.schema import IndexSchemacustom_schema = IndexSchema.from_dict( { # 自定义基本索引规范 "index": { "name": "paul_graham", "prefix": "essay", "key_separator": ":", }, # 自定义被索引的字段 "fields": [ # llamaindex所需的字段 {"type": "tag", "name": "id"}, {"type": "tag", "name": "doc_id"}, {"type": "text", "name": "text"}, # 自定义元数据字段 {"type": "numeric", "name": "updated_at"}, {"type": "tag", "name": "file_name"}, # 用于cohere嵌入的自定义向量字段定义 { "type": "vector", "name": "vector", "attrs": { "dims": 1024, "algorithm": "hnsw", "distance_metric": "cosine", }, }, ], })
custom_schema.index
IndexInfo(name='paul_graham', prefix='essay', key_separator=':', storage_type=<StorageType.HASH: 'hash'>)
custom_schema.fields
{'id': TagField(name='id', type='tag', path=None, attrs=TagFieldAttributes(sortable=False, separator=',', case_sensitive=False, withsuffixtrie=False)), 'doc_id': TagField(name='doc_id', type='tag', path=None, attrs=TagFieldAttributes(sortable=False, separator=',', case_sensitive=False, withsuffixtrie=False)), 'text': TextField(name='text', type='text', path=None, attrs=TextFieldAttributes(sortable=False, weight=1, no_stem=False, withsuffixtrie=False, phonetic_matcher=None)), 'updated_at': NumericField(name='updated_at', type='numeric', path=None, attrs=NumericFieldAttributes(sortable=False)), 'file_name': TagField(name='file_name', type='tag', path=None, attrs=TagFieldAttributes(sortable=False, separator=',', case_sensitive=False, withsuffixtrie=False)), 'vector': HNSWVectorField(name='vector', type='vector', path=None, attrs=HNSWVectorFieldAttributes(dims=1024, algorithm=<VectorIndexAlgorithm.HNSW: 'HNSW'>, datatype=<VectorDataType.FLOAT32: 'FLOAT32'>, distance_metric=<VectorDistanceMetric.COSINE: 'COSINE'>, initial_cap=None, m=16, ef_construction=200, ef_runtime=10, epsilon=0.01))}
了解有关Redis中的模式和索引设计的更多信息。
# from datetime模块 import datetimedef date_to_timestamp(date_string: str) -> int: # 定义日期格式 date_format: str = "%Y-%m-%d" # 返回日期字符串对应的时间戳 return int(datetime.strptime(date_string, date_format).timestamp())# 遍历文档并添加新字段for document in documents: document.metadata["updated_at"] = date_to_timestamp( document.metadata["last_modified_date"] )
# 创建Redis向量存储vector_store = RedisVectorStore( schema=custom_schema, # 提供自定义模式 redis_client=redis_client, overwrite=True, # 覆盖已存在的数据)# 从默认设置创建存储上下文storage_context = StorageContext.from_defaults(vector_store=vector_store)# 从文档和存储上下文构建并加载索引index = VectorStoreIndex.from_documents( documents, storage_context=storage_context)
19:40:05 httpx INFO HTTP Request: POST https://api.cohere.ai/v1/embed "HTTP/1.1 200 OK" 19:40:06 httpx INFO HTTP Request: POST https://api.cohere.ai/v1/embed "HTTP/1.1 200 OK" 19:40:06 httpx INFO HTTP Request: POST https://api.cohere.ai/v1/embed "HTTP/1.1 200 OK" 19:40:06 llama_index.vector_stores.redis.base INFO Added 22 documents to index paul_graham
查询向量存储并根据元数据进行过滤¶
现在我们在Redis中索引了额外的元数据,让我们尝试一些带有过滤器的查询。
from llama_index.core.vector_stores import (
MetadataFilters,
MetadataFilter,
ExactMatchFilter,
)
retriever = index.as_retriever(
similarity_top_k=3,
filters=MetadataFilters(
filters=[
ExactMatchFilter(key="file_name", value="paul_graham_essay.txt"),
MetadataFilter(
key="updated_at",
value=date_to_timestamp("2023-01-01"),
operator=">=",
),
MetadataFilter(
key="text",
value="learn",
operator="text_match",
),
],
condition="and",
),
)
result_nodes = retriever.retrieve("What did the author learn?")
for node in result_nodes:
print(node)
19:40:22 httpx INFO HTTP Request: POST https://api.cohere.ai/v1/embed "HTTP/1.1 200 OK"
19:40:22 llama_index.vector_stores.redis.base INFO Querying index paul_graham with filters ((@file_name:{paul_graham_essay\.txt} @updated_at:[1672549200 +inf]) @text:(learn)) 19:40:22 llama_index.vector_stores.redis.base INFO Found 3 results for query with id ['essay:0df3b734-ecdb-438e-8c90-f21a8c80f552', 'essay:01108c0d-140b-4dcc-b581-c38b7df9251e', 'essay:ced36463-ac36-46b0-b2d7-935c1b38b781'] Node ID: 0df3b734-ecdb-438e-8c90-f21a8c80f552 Text: All that seemed left for philosophy were edge cases that people in other fields felt could safely be ignored. I couldn't have put this into words when I was 18. All I knew at the time was that I kept taking philosophy courses and they kept being boring. So I decided to switch to AI. AI was in the air in the mid 1980s, but there were two things... Score: 0.410 Node ID: 01108c0d-140b-4dcc-b581-c38b7df9251e Text: It was not, in fact, simply a matter of teaching SHRDLU more words. That whole way of doing AI, with explicit data structures representing concepts, was not going to work. Its brokenness did, as so often happens, generate a lot of opportunities to write papers about various band-aids that could be applied to it, but it was never going to get us ... Score: 0.390 Node ID: ced36463-ac36-46b0-b2d7-935c1b38b781 Text: Grad students could take classes in any department, and my advisor, Tom Cheatham, was very easy going. If he even knew about the strange classes I was taking, he never said anything. So now I was in a PhD program in computer science, yet planning to be an artist, yet also genuinely in love with Lisp hacking and working away at On Lisp. In other... Score: 0.389
从Redis中的现有索引进行恢复¶
从索引中进行恢复需要一个Redis连接客户端(或URL),overwrite=False
,并传入之前使用过的相同的模式对象。(可以使用.to_yaml()
将其转储到YAML文件中以方便使用)
custom_schema.to_yaml("paul_graham.yaml")
vector_store = RedisVectorStore(
schema=IndexSchema.from_yaml("paul_graham.yaml"),
redis_client=redis_client,
)
index = VectorStoreIndex.from_vector_store(vector_store=vector_store)
19:40:28 redisvl.index.index INFO Index already exists, not overwriting.
在不久的将来,我们将实现一个便利方法,只需使用索引名称即可加载:
RedisVectorStore.from_existing_index(index_name="paul_graham", redis_client=redis_client)
完全删除文档或索引¶
有时删除文档或整个索引可能会很有用。这可以通过使用 delete
和 delete_index
方法来实现。
document_id = documents[0].doc_id
document_id
'7056f7ba-3513-4ef4-9792-2bd28040aaed'
print("Number of documents before deleting", redis_client.dbsize())
vector_store.delete(document_id)
print("Number of documents after deleting", redis_client.dbsize())
Number of documents before deleting 22 19:40:32 llama_index.vector_stores.redis.base INFO Deleted 22 documents from index paul_graham Number of documents after deleting 0
然而,Redis索引仍然存在(没有关联的文档),以便进行持续的更新。
vector_store.index_exists()
True
# 现在让我们完全删除索引# 这将删除所有文档和索引vector_store.delete_index()
19:40:37 llama_index.vector_stores.redis.base INFO Deleting index paul_graham
print("Number of documents after deleting", redis_client.dbsize())
Number of documents after deleting 0
故障排除¶
如果您得到一个空的查询结果,有一些问题需要检查:
模式¶
与其他向量存储不同,Redis希望用户明确为索引定义模式。这是出于几个原因:
- Redis用于许多用例,包括实时向量搜索,但也用于标准文档存储/检索,缓存,消息传递,发布/订阅,会话管理等。并非所有记录上的属性都需要被索引以供搜索。这在一定程度上是出于效率考虑,部分是为了最小化用户的错误操作。
- 当使用Redis和LlamaIndex时,所有索引模式都必须至少包括以下字段:
id
、doc_id
、text
和vector
。
使用默认模式(假定为OpenAI嵌入)或自定义模式(见上文)实例化您的RedisVectorStore
。
前缀问题¶
Redis希望所有记录都有一个键前缀,将键空间分成“分区”,以供不同应用程序、用例和客户端使用。
确保所选择的prefix
作为索引模式的一部分在您的代码中保持一致(与特定索引相关联)。
要查看您的索引是使用哪个前缀创建的,可以在Redis CLI中运行FT.INFO <您的索引名称>
,然后查看index_definition
=> prefixes
下的内容。
数据与索引¶
Redis将数据集中的记录和索引视为不同的实体。这使您在执行更新、upserts和索引模式迁移时更加灵活。
如果您有一个现有的索引,并希望确保它被删除,可以在Redis CLI中运行FT.DROPINDEX <您的索引名称>
。请注意,这将不会删除您的实际数据,除非您传递DD
参数。
在使用元数据时出现空查询¶
如果在索引已经创建之后向索引添加元数据,然后尝试在该元数据上进行查询,您的查询将返回空结果。
Redis仅在索引创建时对字段进行索引(类似于上文中对前缀的索引)。