如果您在colab上打开这个笔记本,您可能需要安装LlamaIndex 🦙。
%pip install llama-index-readers-wikipedia
%pip install llama-index-llms-azure-openai
%pip install llama-index-graph-stores-nebula
%pip install llama-index-llms-openai
%pip install llama-index-embeddings-azure-openai
!pip install llama-index
让我们首先为Llama指数的基本准备工作做好准备。
OpenAI 是一个人工智能研究实验室,致力于推动人工智能的发展,使其造福全人类。该实验室的使命是确保人工智能的发展对人类的利益最大化。 OpenAI 的目标是开发出一种能够广泛应用的人工智能,同时确保这种技术的安全性和可控性。
# 为OpenAIimport osos.environ["OPENAI_API_KEY"] = "sk-..."import loggingimport syslogging.basicConfig( stream=sys.stdout, level=logging.INFO) # logging.DEBUG 用于更详细的输出# 定义LLMfrom llama_index.llms.openai import OpenAIfrom llama_index.core import SettingsSettings.llm = OpenAI(temperature=0, model="gpt-3.5-turbo")Settings.chunk_size = 512
Azure¶
from llama_index.llms.azure_openai import AzureOpenAIfrom llama_index.embeddings.azure_openai import AzureOpenAIEmbedding# 对于Azure OpenAIapi_key = "<api-key>"azure_endpoint = "https://<your-resource-name>.openai.azure.com/"api_version = "2023-07-01-preview"llm = AzureOpenAI( model="gpt-35-turbo-16k", deployment_name="my-custom-llm", api_key=api_key, azure_endpoint=azure_endpoint, api_version=api_version,)# 您需要部署自己的嵌入模型以及自己的聊天完成模型embed_model = AzureOpenAIEmbedding( model="text-embedding-ada-002", deployment_name="my-custom-embedding", api_key=api_key, azure_endpoint=azure_endpoint, api_version=api_version,)
from llama_index.core import Settings
Settings.llm = llm
Settings.embed_model = embed_model
Settings.chunk_size = 512
准备NebulaGraph¶
在进入下一步创建知识图谱之前,让我们确保我们已经运行了NebulaGraph,并定义了数据模式。
# 使用以下命令创建一个 NebulaGraph(版本 3.5.0 或更新版本)集群:# 选项 0 适用于使用 Docker 的机器:`curl -fsSL nebula-up.siwei.io/install.sh | bash`# 选项 1 适用于桌面版:NebulaGraph Docker 扩展 https://hub.docker.com/extensions/weygu/nebulagraph-dd-ext# 如果不是使用上述方法,可以使用以下命令从 NebulaGraph 控制台创建:# CREATE SPACE llamaindex(vid_type=FIXED_STRING(256), partition_num=1, replica_factor=1);# :sleep 10;# USE llamaindex;# CREATE TAG entity(name string);# CREATE EDGE relationship(relationship string);# :sleep 10;# CREATE TAG INDEX entity_index ON entity(name(256));%pip install ipython-ngql nebula3-pythonos.environ["NEBULA_USER"] = "root"os.environ["NEBULA_PASSWORD"] = "nebula" # 默认为 "nebula"os.environ[ "NEBULA_ADDRESS"] = "127.0.0.1:9669" # 假设已在本地安装了 NebulaGraphspace_name = "llamaindex"edge_types, rel_prop_names = ["relationship"], [ "relationship"] # 默认值,如果从空的知识图谱中创建,则可以省略tags = ["entity"] # 默认值,如果从空的知识图谱中创建,则可以省略
Requirement already satisfied: ipython-ngql in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (0.5)
Requirement already satisfied: nebula3-python in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (3.4.0)
Requirement already satisfied: pandas in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython-ngql) (2.0.3)
Requirement already satisfied: Jinja2 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython-ngql) (3.1.2)
Requirement already satisfied: pytz>=2021.1 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from nebula3-python) (2023.3)
Requirement already satisfied: future>=0.18.0 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from nebula3-python) (0.18.3)
Requirement already satisfied: httplib2>=0.20.0 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from nebula3-python) (0.22.0)
Requirement already satisfied: six>=1.16.0 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from nebula3-python) (1.16.0)
Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from httplib2>=0.20.0->nebula3-python) (3.0.9)
Requirement already satisfied: MarkupSafe>=2.0 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from Jinja2->ipython-ngql) (2.1.3)
Requirement already satisfied: tzdata>=2022.1 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from pandas->ipython-ngql) (2023.3)
Requirement already satisfied: numpy>=1.20.3 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from pandas->ipython-ngql) (1.25.2)
Requirement already satisfied: python-dateutil>=2.8.2 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from pandas->ipython-ngql) (2.8.2)
WARNING: You are using pip version 21.2.4; however, version 23.2.1 is available.
You should consider upgrading via the '/Users/loganmarkewich/llama_index/llama-index/bin/python -m pip install --upgrade pip' command.
Note: you may need to restart the kernel to use updated packages.
准备好使用graph_store作为NebulaGraphStore的StorageContext
from llama_index.core import StorageContext
from llama_index.graph_stores.nebula import NebulaGraphStore
graph_store = NebulaGraphStore(
space_name=space_name,
edge_types=edge_types,
rel_prop_names=rel_prop_names,
tags=tags,
)
storage_context = StorageContext.from_defaults(graph_store=graph_store)
(可选)使用LlamaIndex构建知识图谱¶
在Llama Index和LLM的帮助下,我们可以从给定的文档构建知识图谱。
如果我们已经在NebulaGraphStore上有一个知识图谱,那么这一步可以跳过。
第一步,从维基百科加载“银河护卫队3”电影的数据¶
from llama_index.core import download_loader
from llama_index.readers.wikipedia import WikipediaReader
loader = WikipediaReader()
documents = loader.load_data(
pages=["Guardians of the Galaxy Vol. 3"], auto_suggest=False
)
from llama_index.core import KnowledgeGraphIndex
kg_index = KnowledgeGraphIndex.from_documents(
documents,
storage_context=storage_context,
max_triplets_per_chunk=10,
space_name=space_name,
edge_types=edge_types,
rel_prop_names=rel_prop_names,
tags=tags,
include_embeddings=True,
)
现在我们在名为llamaindex
的NebulaGraph集群上有一个有关《银河护卫队3》电影的知识图谱,让我们稍微玩一下。
# 安装相关的包,密码默认为 nebula%pip install ipython-ngql networkx pyvis%load_ext ngql%ngql --address 127.0.0.1 --port 9669 --user root --password <password>
Requirement already satisfied: ipython-ngql in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (0.5)
Requirement already satisfied: networkx in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (3.1)
Requirement already satisfied: pyvis in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (0.3.2)
Requirement already satisfied: Jinja2 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython-ngql) (3.1.2)
Requirement already satisfied: pandas in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython-ngql) (2.0.3)
Requirement already satisfied: nebula3-python in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython-ngql) (3.4.0)
Requirement already satisfied: jsonpickle>=1.4.1 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from pyvis) (3.0.1)
Requirement already satisfied: ipython>=5.3.0 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from pyvis) (8.10.0)
Requirement already satisfied: backcall in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis) (0.2.0)
Requirement already satisfied: pickleshare in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis) (0.7.5)
Requirement already satisfied: prompt-toolkit<3.1.0,>=3.0.30 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis) (3.0.39)
Requirement already satisfied: appnope in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis) (0.1.3)
Requirement already satisfied: pygments>=2.4.0 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis) (2.15.1)
Requirement already satisfied: traitlets>=5 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis) (5.9.0)
Requirement already satisfied: pexpect>4.3 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis) (4.8.0)
Requirement already satisfied: stack-data in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis) (0.6.2)
Requirement already satisfied: decorator in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis) (5.1.1)
Requirement already satisfied: jedi>=0.16 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis) (0.18.2)
Requirement already satisfied: matplotlib-inline in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis) (0.1.6)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from jedi>=0.16->ipython>=5.3.0->pyvis) (0.8.3)
Requirement already satisfied: MarkupSafe>=2.0 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from Jinja2->ipython-ngql) (2.1.3)
Requirement already satisfied: ptyprocess>=0.5 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from pexpect>4.3->ipython>=5.3.0->pyvis) (0.7.0)
Requirement already satisfied: wcwidth in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from prompt-toolkit<3.1.0,>=3.0.30->ipython>=5.3.0->pyvis) (0.2.6)
Requirement already satisfied: six>=1.16.0 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from nebula3-python->ipython-ngql) (1.16.0)
Requirement already satisfied: pytz>=2021.1 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from nebula3-python->ipython-ngql) (2023.3)
Requirement already satisfied: future>=0.18.0 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from nebula3-python->ipython-ngql) (0.18.3)
Requirement already satisfied: httplib2>=0.20.0 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from nebula3-python->ipython-ngql) (0.22.0)
Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from httplib2>=0.20.0->nebula3-python->ipython-ngql) (3.0.9)
Requirement already satisfied: python-dateutil>=2.8.2 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from pandas->ipython-ngql) (2.8.2)
Requirement already satisfied: numpy>=1.20.3 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from pandas->ipython-ngql) (1.25.2)
Requirement already satisfied: tzdata>=2022.1 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from pandas->ipython-ngql) (2023.3)
Requirement already satisfied: executing>=1.2.0 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from stack-data->ipython>=5.3.0->pyvis) (1.2.0)
Requirement already satisfied: pure-eval in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from stack-data->ipython>=5.3.0->pyvis) (0.2.2)
Requirement already satisfied: asttokens>=2.1.0 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from stack-data->ipython>=5.3.0->pyvis) (2.2.1)
WARNING: You are using pip version 21.2.4; however, version 23.2.1 is available.
You should consider upgrading via the '/Users/loganmarkewich/llama_index/llama-index/bin/python -m pip install --upgrade pip' command.
Note: you may need to restart the kernel to use updated packages.
Connection Pool Created
INFO:nebula3.logger:Get connection to ('127.0.0.1', 9669)
[ERROR]:
'IPythonNGQL' object has no attribute '_decode_value'
Name | |
---|---|
0 | llamaindex |
# 使用Cypher查询一些随机的关系%ngql 使用 llamaindex;%ngql 匹配 ()-[e]->() 返回 e 限制 10
INFO:nebula3.logger:Get connection to ('127.0.0.1', 9669) INFO:nebula3.logger:Get connection to ('127.0.0.1', 9669)
e | |
---|---|
0 | ("A second trailer for the film")-[:relationsh... |
1 | ("Adam McKay")-[:relationship@-442854342936029... |
2 | ("Adam McKay")-[:relationship@8513344855738553... |
3 | ("Asim Chaudhry")-[:relationship@-803614038978... |
4 | ("Bakalova")-[:relationship@-25325064520311626... |
5 | ("Bautista")-[:relationship@-90386029986457371... |
6 | ("Bautista")-[:relationship@-90386029986457371... |
7 | ("Beth Mickle")-[:relationship@716197657641767... |
8 | ("Bradley Cooper")-[:relationship@138630731832... |
9 | ("Bradley Cooper")-[:relationship@838402633192... |
# 绘制结果%ng_draw
nebulagraph_draw.html
查询知识图谱¶
最后,让我们演示如何使用自然语言查询知识图谱!
在这里,我们将利用KnowledgeGraphQueryEngine
,并将NebulaGraphStore
作为storage_context.graph_store
。
from llama_index.core.query_engine import KnowledgeGraphQueryEngine
from llama_index.core import StorageContext
from llama_index.graph_stores.nebula import NebulaGraphStore
query_engine = KnowledgeGraphQueryEngine(
storage_context=storage_context,
llm=llm,
verbose=True,
)
response = query_engine.query(
"Tell me about Peter Quill?",
)
display(Markdown(f"<b>{response}</b>"))
Graph Store Query: ``` MATCH (p:`entity`)-[:relationship]->(m:`entity`) WHERE p.`entity`.`name` == 'Peter Quill' RETURN p.`entity`.`name`; ``` Graph Store Response: {'p.entity.name': ['Peter Quill', 'Peter Quill', 'Peter Quill', 'Peter Quill', 'Peter Quill']} Final Response: Peter Quill is a character in the Marvel Universe. He is the son of Meredith Quill and Ego the Living Planet.
Peter Quill is a character in the Marvel Universe. He is the son of Meredith Quill and Ego the Living Planet.
graph_query = query_engine.generate_query( "告诉我关于彼得·奎尔的信息?",)graph_query = graph_query.replace("WHERE", "\n WHERE").replace( "RETURN", "\nRETURN")display( Markdown( f"""{graph_query}""" ))
MATCH (p:entity
)-[:relationship]->(m:entity
)
WHERE p.entity
.name
== 'Peter Quill'
RETURN p.entity
.name
;
我们可以看到它有助于生成图查询:
MATCH (p:`entity`)-[:relationship]->(e:`entity`)
WHERE p.`entity`.`name` == 'Peter Quill'
RETURN e.`entity`.`name`;
并根据其结果综合出问题:
{'e2.entity.name': ['祖父', 'Gamora的另一个版本', '银河护卫队']}
当然,我们仍然可以查询它!而且这个查询引擎可以成为我们最好的图查询语言学习机器人。 :)
%%ngql
MATCH (p:`entity`)-[e:relationship]->(m:`entity`)
WHERE p.`entity`.`name` == 'Peter Quill'
RETURN p.`entity`.`name`, e.relationship, m.`entity`.`name`;
INFO:nebula3.logger:Get connection to ('127.0.0.1', 9669)
p.entity.name | e.relationship | m.entity.name | |
---|---|---|---|
0 | Peter Quill | would return to the MCU | May 2021 |
1 | Peter Quill | was abducted from Earth | as a child |
2 | Peter Quill | is leader of | Guardians of the Galaxy |
3 | Peter Quill | was raised by | a group of alien thieves and smugglers |
4 | Peter Quill | is half-human | half-Celestial |
并且将查询更改为被渲染
%%ngql
MATCH (p:`entity`)-[e:relationship]->(m:`entity`)
WHERE p.`entity`.`name` == 'Peter Quill'
RETURN p, e, m;
INFO:nebula3.logger:Get connection to ('127.0.0.1', 9669)
p | e | m | |
---|---|---|---|
0 | ("Peter Quill" :entity{name: "Peter Quill"}) | ("Peter Quill")-[:relationship@-84437522554765... | ("May 2021" :entity{name: "May 2021"}) |
1 | ("Peter Quill" :entity{name: "Peter Quill"}) | ("Peter Quill")-[:relationship@-11770408155938... | ("as a child" :entity{name: "as a child"}) |
2 | ("Peter Quill" :entity{name: "Peter Quill"}) | ("Peter Quill")-[:relationship@-79394488349732... | ("Guardians of the Galaxy" :entity{name: "Guar... |
3 | ("Peter Quill" :entity{name: "Peter Quill"}) | ("Peter Quill")-[:relationship@325695233021653... | ("a group of alien thieves and smugglers" :ent... |
4 | ("Peter Quill" :entity{name: "Peter Quill"}) | ("Peter Quill")-[:relationship@555553046209276... | ("half-Celestial" :entity{name: "half-Celestia... |
%ng_draw
nebulagraph_draw.html
这个知识获取查询的结果在渲染的图表中表现得再清晰不过了。