视频数据库检索器¶

RAG：即时搜索和流式传输视频结果 📺¶

VideoDB 是一个无服务器数据库，旨在简化视频内容的存储、搜索、编辑和流式传输。VideoDB通过构建索引和开发查询和浏览视频内容的接口，提供对顺序视频数据的随机访问。更多信息请访问docs.videodb.io。

构建用于文本的RAG管道相对比较简单，这要归功于用于解析、索引和检索文本数据的工具。然而，为视频内容调整RAG模型则面临更大的挑战。视频结合了视觉、听觉和文本元素，需要更多的处理能力和复杂的视频管道。

虽然大型语言模型（LLMs）在文本方面表现出色，但在帮助您消费或创建视频剪辑方面表现不足。VideoDB为您的MP4文件提供了一个复杂的数据库抽象，使您能够在视频数据上使用LLMs。通过VideoDB，您不仅可以分析，还可以立即观看搜索结果的视频流。

在本笔记本中，我们介绍了VideoDBRetriever，这是一个专门设计用于简化视频内容的RAG管道创建的工具，而无需处理复杂的视频基础设施的任何麻烦。

🛠️️ 设置连接¶

需求¶

要连接到VideoDB，只需获取API密钥并创建连接。可以通过设置VIDEO_DB_API_KEY环境变量来实现。您可以从👉🏼VideoDB控制台获取它。（前50次上传免费，无需信用卡！）

从OpenAI平台获取您的OPENAI_API_KEY，用于llama_index响应合成器。

In [ ]:

import os

os.environ["OPENAI_API_KEY"] = ""
os.environ["VIDEO_DB_API_KEY"] = ""

import os os.environ["OPENAI_API_KEY"] = "" os.environ["VIDEO_DB_API_KEY"] = ""

安装依赖项¶

要开始使用，我们需要安装以下软件包：

• llama-index
• llama-index-retrievers-videodb
• videodb

In [ ]:

%pip install llama-index
%pip install videodb

%pip install llama-index %pip install videodb

In [ ]:

%pip install llama-index-retrievers-videodb

%pip install llama-index-retrievers-videodb

数据摄入¶

让我们首先上传一些视频文件。您可以使用任何公共网址、Youtube链接或您系统中的本地文件。前50次上传是免费的！

In [ ]:

from videodb import connect

# 连接到VideoDB
conn = connect()

# 将视频上传到VideoDB中的默认集合
print("上传第一个视频")
video1 = conn.upload(url="https://www.youtube.com/watch?v=lsODSDmY4CY")
print("上传第二个视频")
video2 = conn.upload(url="https://www.youtube.com/watch?v=vZ4kOr38JhY")

from videodb import connect # 连接到VideoDB conn = connect() # 将视频上传到VideoDB中的默认集合 print("上传第一个视频") video1 = conn.upload(url="https://www.youtube.com/watch?v=lsODSDmY4CY") print("上传第二个视频") video2 = conn.upload(url="https://www.youtube.com/watch?v=vZ4kOr38JhY")

• coll = conn.get_collection()：返回默认的集合对象。
• coll.get_videos()：返回集合中所有视频的列表。
• coll.get_video(video_id)：根据给定的video_id返回视频对象。

索引¶

要在视频中搜索特定部分，首先需要对视频进行索引。我们有两种可能的视频索引类型。

• index_spoken_words：索引视频中的口头词语。
• index_scenes：索引视频的视觉内容。（注意：此功能目前仅适用于测试用户，加入我们的Discord获取早期访问权限）https://discord.gg/py9P639jGz

In [ ]:

print("Indexing the videos...")
video1.index_spoken_words()
video2.index_spoken_words()

print("Indexing the videos...") video1.index_spoken_words() video2.index_spoken_words()

Indexing the videos...

100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:39<00:00,  2.56it/s]                                                
100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:39<00:00,  2.51it/s]                                                

查询¶

现在视频已经被索引，我们可以使用 VideoDBRetriever 从 VideoDB 中获取相关节点。

In [ ]:

from llama_index.retrievers.videodb import VideoDBRetriever
from llama_index.core import get_response_synthesizer
from llama_index.core.query_engine import RetrieverQueryEngine

from llama_index.retrievers.videodb import VideoDBRetriever from llama_index.core import get_response_synthesizer from llama_index.core.query_engine import RetrieverQueryEngine

In [ ]:

# 默认情况下，VideoDBRetriever 使用 VideoDB 中的默认集合
retriever = VideoDBRetriever()

# 在搜索结果上使用你的 llama_index response_synthesizer。
response_synthesizer = get_response_synthesizer()

query_engine = RetrieverQueryEngine(
    retriever=retriever,
    response_synthesizer=response_synthesizer,
)

# 默认情况下，VideoDBRetriever 使用 VideoDB 中的默认集合 retriever = VideoDBRetriever() # 在搜索结果上使用你的 llama_index response_synthesizer。 response_synthesizer = get_response_synthesizer() query_engine = RetrieverQueryEngine( retriever=retriever, response_synthesizer=response_synthesizer, )

In [ ]:

# 查询所有上传视频以获取文字回答。
response = query_engine.query("多巴胺是什么？")
print(response)

# 查询所有上传视频以获取文字回答。 response = query_engine.query("多巴胺是什么？") print(response)

Dopamine is a neurotransmitter that plays a key role in various brain functions, including motivation, reward, and pleasure. It is involved in regulating mood, movement, and cognitive function.

In [ ]:

response = query_engine.query("What's the benefit of morning sunlight?")
print(response)

response = query_engine.query("What's the benefit of morning sunlight?") print(response)

Morning sunlight can help trigger a cortisol pulse shift, allowing individuals to capture a morning work block by waking up early and exposing themselves to sunlight. This exposure to morning sunlight, along with brief high-intensity exercise, can assist in adjusting the cortisol levels and potentially enhancing productivity during the early hours of the day.

观看搜索结果的视频流¶

虽然Retriever返回的Nodes是TextNode类型，但它们也具有元数据，可以帮助您观看结果的视频流。您可以使用VideoDB的可编程视频流创建所有节点的编译。您甚至可以轻松地使用音频和图像叠加对其进行修改。

In [ ]:

from videodb import connect, play_stream
from videodb.timeline import Timeline
from videodb.asset import VideoAsset

from videodb import connect, play_stream from videodb.timeline import Timeline from videodb.asset import VideoAsset

In [ ]:

# 创建搜索结果的视频流
conn = connect()
timeline = Timeline(conn)

relevant_nodes = retriever.retrieve("早晨阳光有什么好处？")

for node_obj in relevant_nodes:
    node = node_obj.node
    # 为每个节点创建一个视频资产
    node_asset = VideoAsset(
        asset_id=node.metadata["video_id"],
        start=node.metadata["start"],
        end=node.metadata["end"],
    )
    # 将资产添加到时间轴
    timeline.add_inline(node_asset)

# 为编译后的时间轴生成流
stream_url = timeline.generate_stream()
play_stream(stream_url)

# 创建搜索结果的视频流 conn = connect() timeline = Timeline(conn) relevant_nodes = retriever.retrieve("早晨阳光有什么好处？") for node_obj in relevant_nodes: node = node_obj.node # 为每个节点创建一个视频资产 node_asset = VideoAsset( asset_id=node.metadata["video_id"], start=node.metadata["start"], end=node.metadata["end"], ) # 将资产添加到时间轴 timeline.add_inline(node_asset) # 为编译后的时间轴生成流 stream_url = timeline.generate_stream() play_stream(stream_url)

Out[ ]:

'https://console.videodb.io/player?url=https://stream.videodb.io/v3/published/manifests/9c39c8a9-62a2-4b5e-b15d-8565cc58c8ae.m3u8'

 

配置VideoDBRetriever¶

1. 仅检索一个视频的检索器： 您可以传递视频对象的id，以仅在该视频中搜索。

VideoDBRetriever(video="my_video.id")

2. 不同类型索引的检索器：

# 使用关键字搜索的VideoDBRetriever - 匹配单词和句子的精确出现。它仅支持单个视频。
keyword_retriever = VideoDBRetriever(search_type="keyword", video="my_video.id")

# 使用语义搜索的VideoDBRetriever - 适用于问题答案类型的查询。
semantic_retriever = VideoDBRetriever(search_type="semantic")

# [仅适用于VideoDB的beta用户] 使用场景搜索的VideoDBRetriever - 在视频中搜索视觉信息。
visual_retriever = VideoDBRetriever(search_type="scene")

3. 配置阈值参数：

• result_threshold：是检索器返回的结果数量的阈值；默认值为5
• score_threshold：只有得分高于score_threshold的节点才会被检索器返回；默认值为0.2

custom_retriever = VideoDBRetriever(result_threshold=2, score_threshold=0.5)

查看特定节点¶

要观看每个检索到的节点的流，请直接从VideoDB的video对象中直接生成该部分的流。

In [ ]:

relevant_nodes

relevant_nodes

Out[ ]:

[NodeWithScore(node=TextNode(id_='6ca84002-49df-4091-901d-48248dbe0977', embedding=None, metadata={'collection_id': 'c-33978c87-33e6-4259-9e27-a9edc79be9ad', 'video_id': 'm-f201ff7c-88ec-47ca-938b-a4e968676ba0', 'length': '1496.711837', 'title': 'AMA #1: Leveraging Ultradian Cycles, How to Protect Your Brain, Seed Oils Examined and More', 'start': 906.01, 'end': 974.59}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, text=" So for somebody that wants to learn an immense amount of material, or who has the opportunity to capture another Altradian cycle, the other time where that tends to occur is also early days. So some people, by waking up early and using stimulants like caffeine and hydration or some brief high intensity city exercise, can trigger that cortisol pulse to shift a little bit earlier so that they can capture a morning work block that occurs somewhere, let's say between six and 07:30 a.m. So let's think about our typical person, at least in my example, that's waking up around 07:00 a.m. And then I said, has their first Altradian work cycle really flip on? Because that bump in cortisol around 930 or 10:00 a.m. If that person were, say, to. Set their alarm clock for 05:30 a.m. Then get up, get some artificial light. If the sun isn't out, turn on bright artificial lights. Or if the sun happens to be up that time of year, get some sunlight in your eyes. But irrespective of sunlight, were to get a little bit of brief, high intensity exercise, maybe ten or 15 minutes of skipping rope or even just jumping jacks or go out for a brief jog.", start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n'), score=0.440981567),
 NodeWithScore(node=TextNode(id_='2244fd64-121e-4699-ba36-f0f6a110750f', embedding=None, metadata={'collection_id': 'c-33978c87-33e6-4259-9e27-a9edc79be9ad', 'video_id': 'm-eae54005-b5ca-44f1-9c31-fcdb2f1db56a', 'length': '1830.498685', 'title': 'AMA #2: Improve Sleep, Reduce Sugar Cravings, Optimal Protein Intake, Stretching Frequency & More', 'start': 899.772, 'end': 977.986}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, text=" Because the study, as far as I know, has not been done. Whether or not doing resistance training or some other type of exercise would have led to the same effect. Although I have to imagine that if it's moderately intense to intense resistance training, provided it's done far enough away from going to sleep right prior to 6 hours before sleep, that one ought to see the same effects, although that was not a condition in this study. But it's a very nice study. They looked at everything from changes in core body temperature to caloric expenditure. They didn't see huge changes in core body temperature changes, so that couldn't explain the effect. It really appears that the major effect of improving slow wave sleep was due to something in changing the fine structure of the brainwaves that occur during slow wave sleep. In fact, and this is an important point. The subjects in this study did not report subjectively feeling that much better from their sleep. So you might say, well then, why would I even want to bother? However, it's well known that getting sufficient slow wave sleep is important not just for repair, excuse me, for repair of bodily tissues, but also for repair of brain tissues and repair and washout of debris in the brain. And that debris is known to lead to things like dementia.", start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n'), score=0.282342136)]

In [ ]:

from videodb import connect

# retriever = VideoDBRetriever()
# relevant_nodes = retriever.retrieve("What is Dopamine?")

video_node = relevant_nodes[0].node
conn = connect()
coll = conn.get_collection()

video = coll.get_video(video_node.metadata["video_id"])
start = video_node.metadata["start"]
end = video_node.metadata["end"]

stream_url = video.generate_stream(timeline=[(start, end)])
play_stream(stream_url)

from videodb import connect # retriever = VideoDBRetriever() # relevant_nodes = retriever.retrieve("What is Dopamine?") video_node = relevant_nodes[0].node conn = connect() coll = conn.get_collection() video = coll.get_video(video_node.metadata["video_id"]) start = video_node.metadata["start"] end = video_node.metadata["end"] stream_url = video.generate_stream(timeline=[(start, end)]) play_stream(stream_url)

Out[ ]:

'https://console.videodb.io/player?url=https://stream.videodb.io/v3/published/manifests/b7201145-7302-4ec5-b87c-d1a4c6592f69.m3u8'

🧹 清理工作¶

In [ ]:

video1.delete()
video2.delete()

video1.delete() video2.delete()

👨‍👩‍👧‍👦 支持与社区¶

利用自动化和基于人工智能的内容理解能力，VideoDB 提供了无限的创作和再利用内容的可能性。

如果您有任何问题或反馈，请随时与我们联系 🙌🏼

• Discord
• GitHub
• VideoDB
• 电子邮件