使用LlamaIndex进行Chroma多模态演示¶
Chroma是一个以人工智能为基础的开源向量数据库,专注于开发者的生产力和幸福感。Chroma的许可证是Apache 2.0。
Chroma是完全类型化、经过充分测试和充分文档化的。
使用以下命令安装Chroma:
pip install chromadb
Chroma以各种模式运行。请参见下面的示例,每个示例都与LangChain集成。
内存中
- 在Python脚本或Jupyter笔记本中带持久性的内存中
- 在脚本或笔记本中保存/加载到磁盘在Docker容器中
- 作为在本地计算机或云中运行的服务器
与任何其他数据库一样,您可以:
.add
.get
.update
.upsert
.delete
.peek
- 和
.query
运行相似性搜索。
在文档中查看完整的文档。
基本示例¶
在这个基本示例中,我们将使用一个 Paul Graham 的文章,将其分成块,使用开源嵌入模型进行嵌入,加载到 Chroma 中,然后进行查询。
如果您在Colab上打开这个笔记本,您可能需要安装LlamaIndex 🦙。
%pip install llama-index-vector-stores-qdrant
%pip install llama-index-embeddings-huggingface
%pip install llama-index-vector-stores-chroma
!pip install llama-index
创建色度指数¶
!pip install llama-index chromadb --quiet
!pip install chromadb==0.4.17
!pip install sentence-transformers
!pip install pydantic==1.10.11
!pip install open-clip-torch
# 导入from llama_index.core import VectorStoreIndex,SimpleDirectoryReaderfrom llama_index.vector_stores.chroma import ChromaVectorStorefrom llama_index.core import StorageContextfrom llama_index.embeddings.huggingface import HuggingFaceEmbeddingfrom IPython.display import Markdown,displayfrom chromadb
# 设置OpenAIimport osimport openaiOPENAI_API_KEY = ""openai.api_key = OPENAI_API_KEYos.environ["OPENAI_API_KEY"] = OPENAI_API_KEY
从维基百科下载图片和文本¶
在这个项目中,我们将学习如何使用Python从维基百科下载页面的图片和文本内容。我们将使用wikipedia
库来获取页面的文本内容,并使用requests
库来下载页面中的图片。
我们将首先安装所需的库,然后编写Python代码来实现这一功能。
import requests
def get_wikipedia_images(title):
response = requests.get(
"https://en.wikipedia.org/w/api.php",
params={
"action": "query",
"format": "json",
"titles": title,
"prop": "imageinfo",
"iiprop": "url|dimensions|mime",
"generator": "images",
"gimlimit": "50",
},
).json()
image_urls = []
for page in response["query"]["pages"].values():
if page["imageinfo"][0]["url"].endswith(".jpg") or page["imageinfo"][
0
]["url"].endswith(".png"):
image_urls.append(page["imageinfo"][0]["url"])
return image_urls
from pathlib import Pathimport urllib.requestimage_uuid = 0MAX_IMAGES_PER_WIKI = 20wiki_titles = { "Tesla Model X", "Pablo Picasso", "Rivian", "The Lord of the Rings", "The Matrix", "The Simpsons",}data_path = Path("mixed_wiki")if not data_path.exists(): Path.mkdir(data_path)for title in wiki_titles: response = requests.get( "https://en.wikipedia.org/w/api.php", params={ "action": "query", "format": "json", "titles": title, "prop": "extracts", "explaintext": True, }, ).json() page = next(iter(response["query"]["pages"].values())) wiki_text = page["extract"] with open(data_path / f"{title}.txt", "w") as fp: fp.write(wiki_text) images_per_wiki = 0 try: # page_py = wikipedia.page(title) list_img_urls = get_wikipedia_images(title) # print(list_img_urls) for url in list_img_urls: if url.endswith(".jpg") or url.endswith(".png"): image_uuid += 1 # image_file_name = title + "_" + url.split("/")[-1] urllib.request.urlretrieve( url, data_path / f"{image_uuid}.jpg" ) images_per_wiki += 1 # Limit the number of images downloaded per wiki page to 15 if images_per_wiki > MAX_IMAGES_PER_WIKI: break except: print(str(Exception("No images found for Wikipedia page: ")) + title) continue
设置嵌入模型¶
from chromadb.utils.embedding_functions import OpenCLIPEmbeddingFunction# 设置默认的文本和图像嵌入函数embedding_function = OpenCLIPEmbeddingFunction()
/Users/haotianzhang/llama_index/venv/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm
使用LlamaIndex构建Chroma多模态索引¶
from llama_index.core.indices import MultiModalVectorStoreIndexfrom llama_index.vector_stores.qdrant import QdrantVectorStorefrom llama_index.core import SimpleDirectoryReader, StorageContextfrom chromadb.utils.data_loaders import ImageLoaderimage_loader = ImageLoader()# 创建客户端和新的集合chroma_client = chromadb.EphemeralClient()chroma_collection = chroma_client.create_collection( "multimodal_collection", embedding_function=embedding_function, data_loader=image_loader,)# 加载文档documents = SimpleDirectoryReader("./mixed_wiki/").load_data()# 设置ChromaVectorStore并加载数据vector_store = ChromaVectorStore(chroma_collection=chroma_collection)storage_context = StorageContext.from_defaults(vector_store=vector_store)index = VectorStoreIndex.from_documents( documents, storage_context=storage_context,)
从多模态索引中检索结果¶
retriever = index.as_retriever(similarity_top_k=50)
retrieval_results = retriever.retrieve("Picasso famous paintings")
# 打印检索结果from llama_index.core.schema import ImageNodefrom llama_index.core.response.notebook_utils import ( display_source_node, display_image_uris,)image_results = []MAX_RES = 5cnt = 0for r in retrieval_results: if isinstance(r.node, ImageNode): image_results.append(r.node.metadata["file_path"]) else: if cnt < MAX_RES: display_source_node(r) cnt += 1display_image_uris(image_results, [3, 3], top_k=2)
Node ID: 13adcbba-fe8b-4d51-9139-fb1c55ffc6be
Similarity: 0.774399292477267
Text: == Artistic legacy ==
Picasso's influence was and remains immense and widely acknowledged by his ...
Node ID: 4100593e-6b6a-4b5f-8384-98d1c2468204
Similarity: 0.7695965506408678
Text: === Later works to final years: 1949–1973 ===
Picasso was one of 250 sculptors who exhibited in t...
Node ID: aeed9d43-f9c5-42a9-a7b9-1a3c005e3745
Similarity: 0.7693110304140338
Text: Pablo Ruiz Picasso (25 October 1881 – 8 April 1973) was a Spanish painter, sculptor, printmaker, ...
Node ID: 5a6613b6-b599-4e40-92f2-231e10ed54f6
Similarity: 0.7656537748231977
Text: === The Basel vote ===
In the 1940s, a Swiss insurance company based in Basel had bought two pain...
Node ID: cc17454c-030d-4f86-a12e-342d0582f4d3
Similarity: 0.7639671751819532
Text: == Style and technique ==
Picasso was exceptionally prolific throughout his long lifetime. At hi...