Skip to main content
Open In ColabOpen on GitHub

LlamaEdge

LlamaEdge 允许你与 GGUF 格式的LLMs进行本地和通过聊天服务的对话。

  • LlamaEdgeChatService 为开发者提供了一个与OpenAI API兼容的服务，通过HTTP请求与LLMs进行聊天。

  • LlamaEdgeChatLocal 使开发者能够在本地与LLMs聊天（即将推出）。

无论是 LlamaEdgeChatService 还是 LlamaEdgeChatLocal，它们都在由 WasmEdge Runtime 驱动的基础设施上运行，该基础设施为LLM推理任务提供了一个轻量级且可移植的WebAssembly容器环境。

通过API服务进行聊天​

LlamaEdgeChatService 运行在 llama-api-server 上。按照 llama-api-server 快速入门 中的步骤，您可以托管自己的 API 服务，这样您就可以在任何有互联网的设备上与任何您喜欢的模型进行聊天。

from langchain_community.chat_models.llama_edge import LlamaEdgeChatService
from langchain_core.messages import HumanMessage, SystemMessage

与非流式模式下的LLMs聊天​

# service url
service_url = "https://b008-54-186-154-209.ngrok-free.app"

# create wasm-chat service instance
chat = LlamaEdgeChatService(service_url=service_url)

# create message sequence
system_message = SystemMessage(content="You are an AI assistant")
user_message = HumanMessage(content="What is the capital of France?")
messages = [system_message, user_message]

# chat with wasm-chat service
response = chat.invoke(messages)

print(f"[Bot] {response.content}")
[Bot] Hello! The capital of France is Paris.

与LLMs在流模式下的聊天​

# service url
service_url = "https://b008-54-186-154-209.ngrok-free.app"

# create wasm-chat service instance
chat = LlamaEdgeChatService(service_url=service_url, streaming=True)

# create message sequence
system_message = SystemMessage(content="You are an AI assistant")
user_message = HumanMessage(content="What is the capital of Norway?")
messages = [
    system_message,
    user_message,
]

output = ""
for chunk in chat.stream(messages):
    # print(chunk.content, end="", flush=True)
    output += chunk.content

print(f"[Bot] {output}")
[Bot]   Hello! I'm happy to help you with your question. The capital of Norway is Oslo.

这个页面有帮助吗？