Memgraph
Memgraph 是一个开源的图数据库,专为动态分析环境优化,并与 Neo4j 兼容。为了查询数据库,Memgraph 使用 Cypher - 这是最广泛采用、完全指定且开放的属性图数据库查询语言。
本笔记本将向您展示如何使用自然语言查询Memgraph以及如何从非结构化数据构建知识图谱。
但首先,请确保设置好一切。
设置
要学习本指南,您需要安装Docker和Python 3.x。
要快速首次运行Memgraph平台(Memgraph数据库 + MAGE库 + Memgraph Lab),请执行以下操作:
在Linux/MacOS上:
curl https://install.memgraph.com | sh
在Windows上:
iwr https://windows.memgraph.com | iex
这两个命令运行一个脚本,该脚本将下载一个Docker Compose文件到您的系统,并在两个独立的容器中构建并启动memgraph-mage
和memgraph-lab
Docker服务。现在您已经启动并运行了Memgraph!在Memgraph文档上阅读更多关于安装过程的信息。
要使用LangChain,请安装并导入所有必要的包。我们将使用包管理器pip,以及--user
标志,以确保正确的权限。如果您已经安装了Python 3.4或更高版本,pip
默认已包含在内。您可以使用以下命令安装所有必需的包:
pip install langchain langchain-openai neo4j --user
你可以在这个笔记本中运行提供的代码块,或者使用一个单独的Python文件来实验Memgraph和LangChain。
自然语言查询
Memgraph与LangChain的集成包括自然语言查询。要使用它,首先进行所有必要的导入。我们将在代码中讨论它们。
首先,实例化 MemgraphGraph
。该对象保存与正在运行的 Memgraph 实例的连接。确保正确设置所有环境变量。
import os
from langchain_community.chains.graph_qa.memgraph import MemgraphQAChain
from langchain_community.graphs import MemgraphGraph
from langchain_core.prompts import PromptTemplate
from langchain_openai import ChatOpenAI
url = os.environ.get("MEMGRAPH_URI", "bolt://localhost:7687")
username = os.environ.get("MEMGRAPH_USERNAME", "")
password = os.environ.get("MEMGRAPH_PASSWORD", "")
graph = MemgraphGraph(
url=url, username=username, password=password, refresh_schema=False
)
refresh_schema
最初设置为 False
,因为数据库中还没有数据,我们希望避免不必要的数据库调用。
填充数据库
要填充数据库,首先确保它是空的。最有效的方法是切换到内存分析存储模式,删除图并返回到内存事务模式。了解更多关于Memgraph的存储模式。
我们将添加到数据库的数据是关于不同平台上可用的各种类型的视频游戏以及与出版商相关的信息。
# Drop graph
graph.query("STORAGE MODE IN_MEMORY_ANALYTICAL")
graph.query("DROP GRAPH")
graph.query("STORAGE MODE IN_MEMORY_TRANSACTIONAL")
# Creating and executing the seeding query
query = """
MERGE (g:Game {name: "Baldur's Gate 3"})
WITH g, ["PlayStation 5", "Mac OS", "Windows", "Xbox Series X/S"] AS platforms,
["Adventure", "Role-Playing Game", "Strategy"] AS genres
FOREACH (platform IN platforms |
MERGE (p:Platform {name: platform})
MERGE (g)-[:AVAILABLE_ON]->(p)
)
FOREACH (genre IN genres |
MERGE (gn:Genre {name: genre})
MERGE (g)-[:HAS_GENRE]->(gn)
)
MERGE (p:Publisher {name: "Larian Studios"})
MERGE (g)-[:PUBLISHED_BY]->(p);
"""
graph.query(query)
[]
注意graph
对象如何持有query
方法。该方法在Memgraph中执行查询,并且MemgraphQAChain
也使用它来查询数据库。
刷新图模式
由于新数据是在Memgraph中创建的,因此需要刷新模式。生成的模式将由MemgraphQAChain
使用,以指导LLM更好地生成Cypher查询。
graph.refresh_schema()
为了熟悉数据并验证更新的图模式,您可以使用以下语句打印它:
print(graph.get_schema)
Node labels and properties (name and type) are:
- labels: (:Platform)
properties:
- name: string
- labels: (:Genre)
properties:
- name: string
- labels: (:Game)
properties:
- name: string
- labels: (:Publisher)
properties:
- name: string
Nodes are connected with the following relationships:
(:Game)-[:HAS_GENRE]->(:Genre)
(:Game)-[:PUBLISHED_BY]->(:Publisher)
(:Game)-[:AVAILABLE_ON]->(:Platform)
查询数据库
要与OpenAI API进行交互,您必须将您的API密钥配置为环境变量。这确保了您的请求得到适当的授权。您可以找到更多关于获取API密钥的信息这里。要配置API密钥,您可以使用Python的os包:
os.environ["OPENAI_API_KEY"] = "your-key-here"
如果您在Jupyter笔记本中运行代码,请运行上述代码片段。
接下来,创建MemgraphQAChain
,它将用于基于您的图数据的问答过程。temperature parameter
设置为零,以确保可预测和一致的答案。您可以将verbose
参数设置为True
,以接收有关查询生成的更详细消息。
chain = MemgraphQAChain.from_llm(
ChatOpenAI(temperature=0),
graph=graph,
model_name="gpt-4-turbo",
allow_dangerous_requests=True,
)
现在你可以开始提问了!
response = chain.invoke("Which platforms is Baldur's Gate 3 available on?")
print(response["result"])
MATCH (:Game{name: "Baldur's Gate 3"})-[:AVAILABLE_ON]->(platform:Platform)
RETURN platform.name
Baldur's Gate 3 is available on PlayStation 5, Mac OS, Windows, and Xbox Series X/S.
response = chain.invoke("Is Baldur's Gate 3 available on Windows?")
print(response["result"])
MATCH (:Game{name: "Baldur's Gate 3"})-[:AVAILABLE_ON]->(:Platform{name: "Windows"})
RETURN "Yes"
Yes, Baldur's Gate 3 is available on Windows.
链式修饰符
要修改您的链的行为并获取更多上下文或附加信息,您可以修改链的参数。
返回直接查询结果
return_direct
修饰符指定是返回执行的Cypher查询的直接结果还是处理后的自然语言响应。
# Return the result of querying the graph directly
chain = MemgraphQAChain.from_llm(
ChatOpenAI(temperature=0),
graph=graph,
return_direct=True,
allow_dangerous_requests=True,
model_name="gpt-4-turbo",
)
response = chain.invoke("Which studio published Baldur's Gate 3?")
print(response["result"])
MATCH (g:Game {name: "Baldur's Gate 3"})-[:PUBLISHED_BY]->(p:Publisher)
RETURN p.name
[{'p.name': 'Larian Studios'}]
返回查询中间步骤
return_intermediate_steps
链修饰符通过包含查询的中间步骤以及初始查询结果来增强返回的响应。
# Return all the intermediate steps of query execution
chain = MemgraphQAChain.from_llm(
ChatOpenAI(temperature=0),
graph=graph,
allow_dangerous_requests=True,
return_intermediate_steps=True,
model_name="gpt-4-turbo",
)
response = chain.invoke("Is Baldur's Gate 3 an Adventure game?")
print(f"Intermediate steps: {response['intermediate_steps']}")
print(f"Final response: {response['result']}")
MATCH (:Game {name: "Baldur's Gate 3"})-[:HAS_GENRE]->(:Genre {name: "Adventure"})
RETURN "Yes"
Intermediate steps: [{'query': 'MATCH (:Game {name: "Baldur\'s Gate 3"})-[:HAS_GENRE]->(:Genre {name: "Adventure"})\nRETURN "Yes"'}, {'context': [{'"Yes"': 'Yes'}]}]
Final response: Yes.
限制查询结果的数量
当您想要限制查询结果的最大数量时,可以使用top_k
修饰符。
# Limit the maximum number of results returned by query
chain = MemgraphQAChain.from_llm(
ChatOpenAI(temperature=0),
graph=graph,
top_k=2,
allow_dangerous_requests=True,
model_name="gpt-4-turbo",
)
response = chain.invoke("What genres are associated with Baldur's Gate 3?")
print(response["result"])
MATCH (:Game {name: "Baldur's Gate 3"})-[:HAS_GENRE]->(g:Genre)
RETURN g.name;
Adventure, Role-Playing Game
高级查询
随着解决方案的复杂性增加,您可能会遇到需要仔细处理的不同用例。确保应用程序的可扩展性对于维持顺畅的用户流程至关重要,避免任何障碍。
让我们再次实例化我们的链,并尝试回答一些用户可能会问的问题。
chain = MemgraphQAChain.from_llm(
ChatOpenAI(temperature=0),
graph=graph,
model_name="gpt-4-turbo",
allow_dangerous_requests=True,
)
response = chain.invoke("Is Baldur's Gate 3 available on PS5?")
print(response["result"])
MATCH (:Game{name: "Baldur's Gate 3"})-[:AVAILABLE_ON]->(:Platform{name: "PS5"})
RETURN "Yes"
I don't know the answer.
生成的Cypher查询看起来不错,但我们没有收到任何响应信息。这说明了在使用LLMs时的一个常见挑战——用户如何表达查询与数据如何存储之间的不一致。在这种情况下,用户感知与实际数据存储之间的差异可能导致不匹配。提示优化,即通过精炼模型的提示来更好地理解这些区别的过程,是解决这个问题的有效方法。通过提示优化,模型在生成精确和相关查询方面的能力得到提高,从而成功检索到所需数据。
提示优化
为了解决这个问题,我们可以调整问答链的初始Cypher提示。这包括向LLM添加关于用户如何引用特定平台的指导,例如在我们的案例中的PS5。我们使用LangChain的PromptTemplate来实现这一点,创建一个修改后的初始提示。然后,这个修改后的提示作为参数提供给我们的改进后的MemgraphQAChain
实例。
MEMGRAPH_GENERATION_TEMPLATE = """Your task is to directly translate natural language inquiry into precise and executable Cypher query for Memgraph database.
You will utilize a provided database schema to understand the structure, nodes and relationships within the Memgraph database.
Instructions:
- Use provided node and relationship labels and property names from the
schema which describes the database's structure. Upon receiving a user
question, synthesize the schema to craft a precise Cypher query that
directly corresponds to the user's intent.
- Generate valid executable Cypher queries on top of Memgraph database.
Any explanation, context, or additional information that is not a part
of the Cypher query syntax should be omitted entirely.
- Use Memgraph MAGE procedures instead of Neo4j APOC procedures.
- Do not include any explanations or apologies in your responses.
- Do not include any text except the generated Cypher statement.
- For queries that ask for information or functionalities outside the direct
generation of Cypher queries, use the Cypher query format to communicate
limitations or capabilities. For example: RETURN "I am designed to generate
Cypher queries based on the provided schema only."
Schema:
{schema}
With all the above information and instructions, generate Cypher query for the
user question.
If the user asks about PS5, Play Station 5 or PS 5, that is the platform called PlayStation 5.
The question is:
{question}"""
MEMGRAPH_GENERATION_PROMPT = PromptTemplate(
input_variables=["schema", "question"], template=MEMGRAPH_GENERATION_TEMPLATE
)
chain = MemgraphQAChain.from_llm(
ChatOpenAI(temperature=0),
cypher_prompt=MEMGRAPH_GENERATION_PROMPT,
graph=graph,
model_name="gpt-4-turbo",
allow_dangerous_requests=True,
)
response = chain.invoke("Is Baldur's Gate 3 available on PS5?")
print(response["result"])
MATCH (:Game{name: "Baldur's Gate 3"})-[:AVAILABLE_ON]->(:Platform{name: "PlayStation 5"})
RETURN "Yes"
Yes, Baldur's Gate 3 is available on PS5.
现在,通过修订后的初始Cypher提示,其中包括平台命名的指导,我们获得了更准确和相关的结果,这些结果更紧密地符合用户查询。
这种方法可以进一步改进您的QA链。您可以轻松地将额外的提示优化数据集成到您的链中,从而提升应用程序的整体用户体验。
构建知识图谱
将非结构化数据转换为结构化数据并不是一件容易或直接的任务。本指南将展示如何利用LLMs来帮助我们完成这一任务,以及如何在Memgraph中构建知识图谱。知识图谱创建后,您可以将其用于GraphRAG应用程序。
从文本构建知识图谱的步骤如下:
- 从文本中提取结构化信息: LLM 用于从文本中以节点和关系的形式提取结构化图信息。
- Storing into Memgraph: 将提取的结构化图信息存储到Memgraph中。
从文本中提取结构化信息
除了设置部分中的所有导入外,还需导入LLMGraphTransformer
和Document
,这些将用于从文本中提取结构化信息。
from langchain_core.documents import Document
from langchain_experimental.graph_transformers import LLMGraphTransformer
以下是关于查尔斯·达尔文(来源)的示例文本,将从中构建知识图谱。
text = """
Charles Robert Darwin was an English naturalist, geologist, and biologist,
widely known for his contributions to evolutionary biology. His proposition that
all species of life have descended from a common ancestor is now generally
accepted and considered a fundamental scientific concept. In a joint
publication with Alfred Russel Wallace, he introduced his scientific theory that
this branching pattern of evolution resulted from a process he called natural
selection, in which the struggle for existence has a similar effect to the
artificial selection involved in selective breeding. Darwin has been
described as one of the most influential figures in human history and was
honoured by burial in Westminster Abbey.
"""
下一步是从所需的LLM初始化LLMGraphTransformer
并将文档转换为图结构。
llm = ChatOpenAI(temperature=0, model_name="gpt-4-turbo")
llm_transformer = LLMGraphTransformer(llm=llm)
documents = [Document(page_content=text)]
graph_documents = llm_transformer.convert_to_graph_documents(documents)
在底层,LLM从文本中提取重要实体,并将它们作为节点和关系的列表返回。以下是它的样子:
print(graph_documents)
[GraphDocument(nodes=[Node(id='Charles Robert Darwin', type='Person', properties={}), Node(id='English', type='Nationality', properties={}), Node(id='Naturalist', type='Profession', properties={}), Node(id='Geologist', type='Profession', properties={}), Node(id='Biologist', type='Profession', properties={}), Node(id='Evolutionary Biology', type='Field', properties={}), Node(id='Common Ancestor', type='Concept', properties={}), Node(id='Scientific Concept', type='Concept', properties={}), Node(id='Alfred Russel Wallace', type='Person', properties={}), Node(id='Natural Selection', type='Concept', properties={}), Node(id='Selective Breeding', type='Concept', properties={}), Node(id='Westminster Abbey', type='Location', properties={})], relationships=[Relationship(source=Node(id='Charles Robert Darwin', type='Person', properties={}), target=Node(id='English', type='Nationality', properties={}), type='NATIONALITY', properties={}), Relationship(source=Node(id='Charles Robert Darwin', type='Person', properties={}), target=Node(id='Naturalist', type='Profession', properties={}), type='PROFESSION', properties={}), Relationship(source=Node(id='Charles Robert Darwin', type='Person', properties={}), target=Node(id='Geologist', type='Profession', properties={}), type='PROFESSION', properties={}), Relationship(source=Node(id='Charles Robert Darwin', type='Person', properties={}), target=Node(id='Biologist', type='Profession', properties={}), type='PROFESSION', properties={}), Relationship(source=Node(id='Charles Robert Darwin', type='Person', properties={}), target=Node(id='Evolutionary Biology', type='Field', properties={}), type='CONTRIBUTION', properties={}), Relationship(source=Node(id='Common Ancestor', type='Concept', properties={}), target=Node(id='Scientific Concept', type='Concept', properties={}), type='BASIS', properties={}), Relationship(source=Node(id='Charles Robert Darwin', type='Person', properties={}), target=Node(id='Alfred Russel Wallace', type='Person', properties={}), type='COLLABORATION', properties={}), Relationship(source=Node(id='Natural Selection', type='Concept', properties={}), target=Node(id='Selective Breeding', type='Concept', properties={}), type='COMPARISON', properties={}), Relationship(source=Node(id='Charles Robert Darwin', type='Person', properties={}), target=Node(id='Westminster Abbey', type='Location', properties={}), type='BURIAL', properties={})], source=Document(metadata={}, page_content='\n Charles Robert Darwin was an English naturalist, geologist, and biologist,\n widely known for his contributions to evolutionary biology. His proposition that\n all species of life have descended from a common ancestor is now generally\n accepted and considered a fundamental scientific concept. In a joint\n publication with Alfred Russel Wallace, he introduced his scientific theory that\n this branching pattern of evolution resulted from a process he called natural\n selection, in which the struggle for existence has a similar effect to the\n artificial selection involved in selective breeding. Darwin has been\n described as one of the most influential figures in human history and was\n honoured by burial in Westminster Abbey.\n'))]
存储到Memgraph
一旦你准备好了GraphDocument
格式的数据,即节点和关系,你可以使用add_graph_documents
方法将其导入到Memgraph中。该方法将graph_documents
列表转换为需要在Memgraph中执行的适当Cypher查询。完成后,知识图将存储在Memgraph中。
# Empty the database
graph.query("STORAGE MODE IN_MEMORY_ANALYTICAL")
graph.query("DROP GRAPH")
graph.query("STORAGE MODE IN_MEMORY_TRANSACTIONAL")
# Create KG
graph.add_graph_documents(graph_documents)
这是在Memgraph Lab中图表的样子(请在localhost:3000
上查看):
如果你尝试了这个操作并得到了不同的图表,这是预期行为。图表构建过程是非确定性的,因为用于从非结构化数据生成节点和关系的LLM是非确定性的。
附加选项
此外,您可以根据需要灵活定义特定类型的节点和关系进行提取。
llm_transformer_filtered = LLMGraphTransformer(
llm=llm,
allowed_nodes=["Person", "Nationality", "Concept"],
allowed_relationships=["NATIONALITY", "INVOLVED_IN", "COLLABORATES_WITH"],
)
graph_documents_filtered = llm_transformer_filtered.convert_to_graph_documents(
documents
)
print(f"Nodes:{graph_documents_filtered[0].nodes}")
print(f"Relationships:{graph_documents_filtered[0].relationships}")
Nodes:[Node(id='Charles Robert Darwin', type='Person', properties={}), Node(id='English', type='Nationality', properties={}), Node(id='Evolutionary Biology', type='Concept', properties={}), Node(id='Natural Selection', type='Concept', properties={}), Node(id='Alfred Russel Wallace', type='Person', properties={})]
Relationships:[Relationship(source=Node(id='Charles Robert Darwin', type='Person', properties={}), target=Node(id='English', type='Nationality', properties={}), type='NATIONALITY', properties={}), Relationship(source=Node(id='Charles Robert Darwin', type='Person', properties={}), target=Node(id='Evolutionary Biology', type='Concept', properties={}), type='INVOLVED_IN', properties={}), Relationship(source=Node(id='Charles Robert Darwin', type='Person', properties={}), target=Node(id='Natural Selection', type='Concept', properties={}), type='INVOLVED_IN', properties={}), Relationship(source=Node(id='Charles Robert Darwin', type='Person', properties={}), target=Node(id='Alfred Russel Wallace', type='Person', properties={}), type='COLLABORATES_WITH', properties={})]
在这种情况下,图表将如下所示:
您的图表还可以在所有节点上具有__Entity__
标签,这些标签将被索引以便更快地检索。
# Drop graph
graph.query("STORAGE MODE IN_MEMORY_ANALYTICAL")
graph.query("DROP GRAPH")
graph.query("STORAGE MODE IN_MEMORY_TRANSACTIONAL")
# Store to Memgraph with Entity label
graph.add_graph_documents(graph_documents, baseEntityLabel=True)
这是图表的样子:
还有一个选项可以包含在图中获取的信息的来源。要做到这一点,将include_source
设置为True
,然后源文档将被存储,并使用MENTIONS
关系将其链接到图中的节点。
# Drop graph
graph.query("STORAGE MODE IN_MEMORY_ANALYTICAL")
graph.query("DROP GRAPH")
graph.query("STORAGE MODE IN_MEMORY_TRANSACTIONAL")
# Store to Memgraph with source included
graph.add_graph_documents(graph_documents, include_source=True)
构建的图将如下所示:
注意源内容是如何存储的,并且由于文档没有任何id
,生成了id
属性。
你可以同时拥有__Entity__
标签和文档源。但是,请注意两者都会占用内存,特别是由于内容的长字符串,包含源会占用更多内存。
最后,您可以查询知识图谱,如前一节所述:
chain = MemgraphQAChain.from_llm(
ChatOpenAI(temperature=0),
graph=graph,
model_name="gpt-4-turbo",
allow_dangerous_requests=True,
)
print(chain.invoke("Who Charles Robert Darwin collaborated with?")["result"])
MATCH (:Person {id: "Charles Robert Darwin"})-[:COLLABORATION]->(collaborator)
RETURN collaborator;
Alfred Russel Wallace