使用CLIP嵌入和GPT4V进行图像检索¶
在这个笔记本中,我们将展示如何使用LlamaIndex和GPT4-V构建图像到图像的检索。
LlamaIndex图像到图像检索
- 图像嵌入索引:来自OpenAI的CLIP图像嵌入
框架:LlamaIndex
步骤:
- 从维基百科页面下载文本、图像和pdf原始文件
- 构建文本和图像的多模态索引和向量存储
- 使用多模态检索器根据图像查询检索相关图像
- 使用GPT4V推理输入图像和检索到的图像之间的相关性
In [ ]:
Copied!
%pip install llama-index-multi-modal-llms-openai
%pip install llama-index-vector-stores-qdrant
%pip install llama-index-multi-modal-llms-openai
%pip install llama-index-vector-stores-qdrant
In [ ]:
Copied!
%pip install llama_index ftfy regex tqdm
%pip install git+https://github.com/openai/CLIP.git
%pip install torch torchvision
%pip install matplotlib scikit-image
%pip install -U qdrant_client
%pip install llama_index ftfy regex tqdm
%pip install git+https://github.com/openai/CLIP.git
%pip install torch torchvision
%pip install matplotlib scikit-image
%pip install -U qdrant_client
In [ ]:
Copied!
import os
OPENAI_API_KEY = "sk-"
os.environ["OPENAI_API_KEY"] = OPENAI_API_KEY
import os
OPENAI_API_KEY = "sk-"
os.environ["OPENAI_API_KEY"] = OPENAI_API_KEY
从维基百科下载图片和文本¶
在这个项目中,我们将学习如何使用Python从维基百科下载页面的图片和文本内容。我们将使用wikipedia-api
库来获取页面的文本内容,并使用requests
库来下载页面中的图片。
我们将首先安装所需的库,然后编写Python代码来实现这一功能。
In [ ]:
Copied!
import wikipediaimport urllib.requestfrom pathlib import Pathimage_path = Path("mixed_wiki")image_uuid = 0# image_metadata_dict 存储包括图像uuid、文件名和路径在内的图像元数据image_metadata_dict = {}MAX_IMAGES_PER_WIKI = 30wiki_titles = [ "Vincent van Gogh", "San Francisco", "Batman", "iPhone", "Tesla Model S", "BTS band",]# 仅创建用于图像的文件夹if not image_path.exists(): Path.mkdir(image_path)# 为维基页面下载图像# 为每个图像分配UUIDfor title in wiki_titles: images_per_wiki = 0 print(title) try: page_py = wikipedia.page(title) list_img_urls = page_py.images for url in list_img_urls: if url.endswith(".jpg") or url.endswith(".png"): image_uuid += 1 image_file_name = title + "_" + url.split("/")[-1] # img_path 可能是指向未来原始图像文件的s3路径 image_metadata_dict[image_uuid] = { "filename": image_file_name, "img_path": "./" + str(image_path / f"{image_uuid}.jpg"), } urllib.request.urlretrieve( url, image_path / f"{image_uuid}.jpg" ) images_per_wiki += 1 # 限制每个维基页面下载的图像数量为15 if images_per_wiki > MAX_IMAGES_PER_WIKI: break except: print(str(Exception("未找到维基百科页面的图像:")) + title) continue
import wikipediaimport urllib.requestfrom pathlib import Pathimage_path = Path("mixed_wiki")image_uuid = 0# image_metadata_dict 存储包括图像uuid、文件名和路径在内的图像元数据image_metadata_dict = {}MAX_IMAGES_PER_WIKI = 30wiki_titles = [ "Vincent van Gogh", "San Francisco", "Batman", "iPhone", "Tesla Model S", "BTS band",]# 仅创建用于图像的文件夹if not image_path.exists(): Path.mkdir(image_path)# 为维基页面下载图像# 为每个图像分配UUIDfor title in wiki_titles: images_per_wiki = 0 print(title) try: page_py = wikipedia.page(title) list_img_urls = page_py.images for url in list_img_urls: if url.endswith(".jpg") or url.endswith(".png"): image_uuid += 1 image_file_name = title + "_" + url.split("/")[-1] # img_path 可能是指向未来原始图像文件的s3路径 image_metadata_dict[image_uuid] = { "filename": image_file_name, "img_path": "./" + str(image_path / f"{image_uuid}.jpg"), } urllib.request.urlretrieve( url, image_path / f"{image_uuid}.jpg" ) images_per_wiki += 1 # 限制每个维基页面下载的图像数量为15 if images_per_wiki > MAX_IMAGES_PER_WIKI: break except: print(str(Exception("未找到维基百科页面的图像:")) + title) continue
从维基百科绘制图像¶
这个Python脚本从维基百科下载图像并绘制它们。
In [ ]:
Copied!
from PIL import Image
import matplotlib.pyplot as plt
import os
image_paths = []
for img_path in os.listdir("./mixed_wiki"):
image_paths.append(str(os.path.join("./mixed_wiki", img_path)))
def plot_images(image_paths):
images_shown = 0
plt.figure(figsize=(16, 9))
for img_path in image_paths:
if os.path.isfile(img_path):
image = Image.open(img_path)
plt.subplot(3, 3, images_shown + 1)
plt.imshow(image)
plt.xticks([])
plt.yticks([])
images_shown += 1
if images_shown >= 9:
break
plot_images(image_paths)
from PIL import Image
import matplotlib.pyplot as plt
import os
image_paths = []
for img_path in os.listdir("./mixed_wiki"):
image_paths.append(str(os.path.join("./mixed_wiki", img_path)))
def plot_images(image_paths):
images_shown = 0
plt.figure(figsize=(16, 9))
for img_path in image_paths:
if os.path.isfile(img_path):
image = Image.open(img_path)
plt.subplot(3, 3, images_shown + 1)
plt.imshow(image)
plt.xticks([])
plt.yticks([])
images_shown += 1
if images_shown >= 9:
break
plot_images(image_paths)
/Users/haotianzhang/llama_index/venv/lib/python3.11/site-packages/PIL/Image.py:3157: DecompressionBombWarning: Image size (101972528 pixels) exceeds limit of 89478485 pixels, could be decompression bomb DOS attack. warnings.warn(
构建多模态索引和向量存储,以索引维基百科中的文本和图像¶
In [ ]:
Copied!
from llama_index.core.indices import MultiModalVectorStoreIndexfrom llama_index.vector_stores.qdrant import QdrantVectorStorefrom llama_index.core import SimpleDirectoryReader, StorageContextimport qdrant_clientfrom llama_index.core import SimpleDirectoryReader# 创建一个本地的Qdrant向量存储client = qdrant_client.QdrantClient(path="qdrant_img_db")text_store = QdrantVectorStore( client=client, collection_name="text_collection")image_store = QdrantVectorStore( client=client, collection_name="image_collection")storage_context = StorageContext.from_defaults( vector_store=text_store, image_store=image_store)# 创建MultiModal索引documents = SimpleDirectoryReader("./mixed_wiki/").load_data()index = MultiModalVectorStoreIndex.from_documents( documents, storage_context=storage_context,)
from llama_index.core.indices import MultiModalVectorStoreIndexfrom llama_index.vector_stores.qdrant import QdrantVectorStorefrom llama_index.core import SimpleDirectoryReader, StorageContextimport qdrant_clientfrom llama_index.core import SimpleDirectoryReader# 创建一个本地的Qdrant向量存储client = qdrant_client.QdrantClient(path="qdrant_img_db")text_store = QdrantVectorStore( client=client, collection_name="text_collection")image_store = QdrantVectorStore( client=client, collection_name="image_collection")storage_context = StorageContext.from_defaults( vector_store=text_store, image_store=image_store)# 创建MultiModal索引documents = SimpleDirectoryReader("./mixed_wiki/").load_data()index = MultiModalVectorStoreIndex.from_documents( documents, storage_context=storage_context,)
绘制输入的查询图像¶
In [ ]:
Copied!
input_image = "./mixed_wiki/2.jpg"
plot_images([input_image])
input_image = "./mixed_wiki/2.jpg"
plot_images([input_image])
从多模态索引中检索图像给定图像查询¶
1. 图像到图像检索结果¶
In [ ]:
Copied!
# 生成文本检索结果retriever_engine = index.as_retriever(image_similarity_top_k=4)# 从GPT4V响应中检索更多信息retrieval_results = retriever_engine.image_to_image_retrieve( "./mixed_wiki/2.jpg")retrieved_images = []for res in retrieval_results: retrieved_images.append(res.node.metadata["file_path"])# 移除第一个检索到的图像,因为它是输入图像# 由于输入图像将获得最高的相似度分数plot_images(retrieved_images[1:])
# 生成文本检索结果retriever_engine = index.as_retriever(image_similarity_top_k=4)# 从GPT4V响应中检索更多信息retrieval_results = retriever_engine.image_to_image_retrieve( "./mixed_wiki/2.jpg")retrieved_images = []for res in retrieval_results: retrieved_images.append(res.node.metadata["file_path"])# 移除第一个检索到的图像,因为它是输入图像# 由于输入图像将获得最高的相似度分数plot_images(retrieved_images[1:])
2. GPT4V根据输入图像推理检索到的图像¶
In [ ]:
Copied!
from llama_index.multi_modal_llms.openai import OpenAIMultiModalfrom llama_index.core import SimpleDirectoryReaderfrom llama_index.core.schema import ImageDocument# 将本地目录放在这里image_documents = [ImageDocument(image_path=input_image)]for res_img in retrieved_images[1:]: image_documents.append(ImageDocument(image_path=res_img))openai_mm_llm = OpenAIMultiModal( model="gpt-4-vision-preview", api_key=OPENAI_API_KEY, max_new_tokens=1500)response = openai_mm_llm.complete( prompt="给定第一张图像作为基础图像,其他图像对应什么?", image_documents=image_documents,)print(response)
from llama_index.multi_modal_llms.openai import OpenAIMultiModalfrom llama_index.core import SimpleDirectoryReaderfrom llama_index.core.schema import ImageDocument# 将本地目录放在这里image_documents = [ImageDocument(image_path=input_image)]for res_img in retrieved_images[1:]: image_documents.append(ImageDocument(image_path=res_img))openai_mm_llm = OpenAIMultiModal( model="gpt-4-vision-preview", api_key=OPENAI_API_KEY, max_new_tokens=1500)response = openai_mm_llm.complete( prompt="给定第一张图像作为基础图像,其他图像对应什么?", image_documents=image_documents,)print(response)
The images you provided appear to be works of art, and although I should not provide specific artist names or titles as they can be seen as identifying works or artists, I will describe each picture and discuss their similarities. 1. The first image displays a style characterized by bold, visible brushstrokes and a vibrant use of color. It features a figure with a tree against a backdrop of a luminous yellow moon and blue sky. The impression is one of dynamic movement and emotion conveyed through color and form. 2. The second image is similar in style, with distinctive brushstrokes and vivid coloration. This painting depicts a landscape of twisting trees and rolling hills under a cloud-filled sky. The energetic application of paint and color connects it to the first image's aesthetic. 3. The third image, again, shares the same painterly characteristics—thick brushstrokes and intense hues. It portrays a man leaning over a table with a bouquet of sunflowers, hinting at a personal, intimate setting. This painting's expressive quality and the bold use of color align it with the first two. 4. The fourth image continues with the same artistic style. This is a landscape featuring hay stacks under a swirling sky with a large, crescent moon. The movement in the sky and the textured field convey a sense of rhythm and evoke a specific mood typical of the other images. All four images showcase a consistent art style that is commonly associated with Post-Impressionism, where the focus is on symbolic content, formal experimentation, and a vivid palette. The distinctive brushwork and color choices suggest that these paintings could be by the same artist or from a similar artistic movement.
使用图像查询引擎¶
在查询引擎中,有几个步骤:
根据输入图像检索相关图像
使用提示文本构建
image_qa_template
将前k个检索到的图像和image_qa_template发送给GPT4V进行回答/合成
In [ ]:
Copied!
from llama_index.multi_modal_llms.openai import OpenAIMultiModal
from llama_index.core import PromptTemplate
qa_tmpl_str = (
"Given the images provided, "
"answer the query.\n"
"Query: {query_str}\n"
"Answer: "
)
qa_tmpl = PromptTemplate(qa_tmpl_str)
openai_mm_llm = OpenAIMultiModal(
model="gpt-4-vision-preview", api_key=OPENAI_API_KEY, max_new_tokens=1500
)
query_engine = index.as_query_engine(
multi_modal_llm=openai_mm_llm, image_qa_template=qa_tmpl
)
query_str = "Tell me more about the relationship between those paintings. "
response = query_engine.image_query("./mixed_wiki/2.jpg", query_str)
from llama_index.multi_modal_llms.openai import OpenAIMultiModal
from llama_index.core import PromptTemplate
qa_tmpl_str = (
"Given the images provided, "
"answer the query.\n"
"Query: {query_str}\n"
"Answer: "
)
qa_tmpl = PromptTemplate(qa_tmpl_str)
openai_mm_llm = OpenAIMultiModal(
model="gpt-4-vision-preview", api_key=OPENAI_API_KEY, max_new_tokens=1500
)
query_engine = index.as_query_engine(
multi_modal_llm=openai_mm_llm, image_qa_template=qa_tmpl
)
query_str = "Tell me more about the relationship between those paintings. "
response = query_engine.image_query("./mixed_wiki/2.jpg", query_str)
In [ ]:
Copied!
print(response)
print(response)
The first image you've provided is of Vincent van Gogh's painting known as "The Sower." This work is emblematic of Van Gogh's interest in the cycles of nature and the life of the rural worker. Painted in 1888, "The Sower" features a large, yellow sun setting in the background, casting a warm glow over the scene, with a foreground that includes a sower going about his work. Van Gogh’s use of vivid colors and dynamic, almost swirling brushstrokes are characteristic of his famous post-impressionistic style. The second image appears to be "The Olive Trees" by Vincent van Gogh. This painting was also created in 1889, and it showcases Van Gogh's expressive use of color and form. The scene depicts a grove of olive trees with rolling hills in the background and a swirling sky, which is highly reminiscent of the style he used in his most famous work, "The Starry Night." "The Olive Trees" series conveys the vitality and movement that Van Gogh saw in the landscape around him while he was staying in the Saint-Rémy-de-Provence asylum. His brushwork is energetic and his colors are layered in a way to give depth and emotion to the scene.