Firestore演示¶

本指南向您展示如何直接使用由Google Firestore支持的DocumentStore抽象。通过将节点放入文档存储中，这使您能够在相同的基础文档存储上定义多个索引，而不是在索引之间复制数据。

如果您在Colab上打开这个笔记本，您可能需要安装LlamaIndex 🦙。

In [ ]:

%pip install llama-index-storage-docstore-firestore
%pip install llama-index-storage-kvstore-firestore
%pip install llama-index-storage-index-store-firestore
%pip install llama-index-llms-openai

%pip install llama-index-storage-docstore-firestore %pip install llama-index-storage-kvstore-firestore %pip install llama-index-storage-index-store-firestore %pip install llama-index-llms-openai

In [ ]:

!pip install llama-index

!pip install llama-index

In [ ]:

import nest_asyncio

nest_asyncio.apply()

import nest_asyncio nest_asyncio.apply()

In [ ]:

import logging
import sys

logging.basicConfig(stream=sys.stdout, level=logging.INFO)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

import logging import sys logging.basicConfig(stream=sys.stdout, level=logging.INFO) logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

In [ ]:

from llama_index.core import SimpleDirectoryReader, StorageContext
from llama_index.core import VectorStoreIndex, SimpleKeywordTableIndex
from llama_index.core import SummaryIndex
from llama_index.core import ComposableGraph
from llama_index.llms.openai import OpenAI
from llama_index.core.response.notebook_utils import display_response
from llama_index.core import Settings

from llama_index.core import SimpleDirectoryReader, StorageContext from llama_index.core import VectorStoreIndex, SimpleKeywordTableIndex from llama_index.core import SummaryIndex from llama_index.core import ComposableGraph from llama_index.llms.openai import OpenAI from llama_index.core.response.notebook_utils import display_response from llama_index.core import Settings

下载数据¶

In [ ]:

!mkdir -p 'data/paul_graham/'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'

!mkdir -p 'data/paul_graham/' !wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'

加载文档¶

In [ ]:

reader = SimpleDirectoryReader("./data/paul_graham/")
documents = reader.load_data()

reader = SimpleDirectoryReader("./data/paul_graham/") documents = reader.load_data()

解析到节点¶

In [ ]:

from llama_index.core.node_parser import SentenceSplitter

nodes = SentenceSplitter().get_nodes_from_documents(documents)

from llama_index.core.node_parser import SentenceSplitter nodes = SentenceSplitter().get_nodes_from_documents(documents)

添加到文档库¶

In [ ]:

from llama_index.storage.kvstore.firestore import FirestoreKVStore
from llama_index.storage.docstore.firestore import FirestoreDocumentStore
from llama_index.storage.index_store.firestore import FirestoreIndexStore

from llama_index.storage.kvstore.firestore import FirestoreKVStore from llama_index.storage.docstore.firestore import FirestoreDocumentStore from llama_index.storage.index_store.firestore import FirestoreIndexStore

In [ ]:

kvstore = FirestoreKVStore()

storage_context = StorageContext.from_defaults(
    docstore=FirestoreDocumentStore(kvstore),
    index_store=FirestoreIndexStore(kvstore),
)

kvstore = FirestoreKVStore() storage_context = StorageContext.from_defaults( docstore=FirestoreDocumentStore(kvstore), index_store=FirestoreIndexStore(kvstore), )

In [ ]:

storage_context.docstore.add_documents(nodes)

storage_context.docstore.add_documents(nodes)

定义多个索引¶

每个索引使用相同的基础节点。

In [ ]:

summary_index = SummaryIndex(nodes, storage_context=storage_context)

summary_index = SummaryIndex(nodes, storage_context=storage_context)

In [ ]:

vector_index = VectorStoreIndex(nodes, storage_context=storage_context)

vector_index = VectorStoreIndex(nodes, storage_context=storage_context)

In [ ]:

keyword_table_index = SimpleKeywordTableIndex(
    nodes, storage_context=storage_context
)

keyword_table_index = SimpleKeywordTableIndex( nodes, storage_context=storage_context )

In [ ]:

# 注意：文档存储仍然具有相同的节点
len(storage_context.docstore.docs)

# 注意：文档存储仍然具有相同的节点 len(storage_context.docstore.docs)

测试保存和加载¶

In [ ]:

# 注意：默认情况下，docstore和index_store会持久化到Firestore中
# 注意：这里只需要将简单的向量存储持久化到磁盘中
storage_context.persist()

# 注意：默认情况下，docstore和index_store会持久化到Firestore中 # 注意：这里只需要将简单的向量存储持久化到磁盘中 storage_context.persist()

In [ ]:

# 记录索引ID
list_id = summary_index.index_id  # 摘要索引ID
vector_id = vector_index.index_id  # 向量索引ID
keyword_id = keyword_table_index.index_id  # 关键词表索引ID

# 记录索引ID list_id = summary_index.index_id # 摘要索引ID vector_id = vector_index.index_id # 向量索引ID keyword_id = keyword_table_index.index_id # 关键词表索引ID

In [ ]:

from llama_index.core import load_index_from_storage

kvstore = FirestoreKVStore()

# 重新创建存储上下文
storage_context = StorageContext.from_defaults(
    docstore=FirestoreDocumentStore(kvstore),
    index_store=FirestoreIndexStore(kvstore),
)

# 加载索引
summary_index = load_index_from_storage(
    storage_context=storage_context, index_id=list_id
)
vector_index = load_index_from_storage(
    storage_context=storage_context, vector_id=vector_id
)
keyword_table_index = load_index_from_storage(
    storage_context=storage_context, keyword_id=keyword_id
)

from llama_index.core import load_index_from_storage kvstore = FirestoreKVStore() # 重新创建存储上下文 storage_context = StorageContext.from_defaults( docstore=FirestoreDocumentStore(kvstore), index_store=FirestoreIndexStore(kvstore), ) # 加载索引 summary_index = load_index_from_storage( storage_context=storage_context, index_id=list_id ) vector_index = load_index_from_storage( storage_context=storage_context, vector_id=vector_id ) keyword_table_index = load_index_from_storage( storage_context=storage_context, keyword_id=keyword_id )

测试一些查询语句¶

In [ ]:

chatgpt = OpenAI(temperature=0, model="gpt-3.5-turbo")
Settings.llm = chatgpt
Settings.chunk_size = 1024

chatgpt = OpenAI(temperature=0, model="gpt-3.5-turbo") Settings.llm = chatgpt Settings.chunk_size = 1024

In [ ]:

query_engine = summary_index.as_query_engine()
list_response = query_engine.query("What is a summary of this document?")

query_engine = summary_index.as_query_engine() list_response = query_engine.query("What is a summary of this document?")

In [ ]:

display_response(list_response)

display_response(list_response)

In [ ]:

query_engine = vector_index.as_query_engine()
vector_response = query_engine.query("What did the author do growing up?")

query_engine = vector_index.as_query_engine() vector_response = query_engine.query("What did the author do growing up?")

In [ ]:

display_response(vector_response)

display_response(vector_response)

In [ ]:

query_engine = keyword_table_index.as_query_engine()
keyword_response = query_engine.query(
    "What did the author do after his time at YC?"
)

query_engine = keyword_table_index.as_query_engine() keyword_response = query_engine.query( "What did the author do after his time at YC?" )

In [ ]:

display_response(keyword_response)

display_response(keyword_response)