跳至主内容

Webhooks

warning
  • 此功能仍然处于试验阶段,未来版本可能会发生更改。
  • 文件后端不支持 webhooks。只有 SQL 后端支持 webhooks。
  • 只有 OSS MLflow 支持 webhooks。Databricks 或其他托管的 MLflow 服务可能不支持此功能。

概览

MLflow webhooks 可在模型注册表中发生特定事件时启用实时通知。当您注册模型、创建新版本或修改标签和别名时,MLflow 可以自动向您指定的端点发送 HTTP POST 请求。这使得与 CI/CD 管道、通知系统和其他外部服务的无缝集成成为可能。

主要特性

  • 实时通知 用于模型注册表事件
  • HMAC 签名验证 用于安全的 webhook 传递
  • 多种事件类型 包括模型创建、版本管理和标记
  • 内置测试 用于验证 webhook 连接

支持的事件

MLflow webhooks 支持以下模型注册事件:

事件描述有效负载模式
registered_model.createdTriggered when a new registered model is createdRegisteredModelCreatedPayload
model_version.createdTriggered when a new model version is createdModelVersionCreatedPayload
model_version_tag.setTriggered when a tag is set on a model versionModelVersionTagSetPayload
model_version_tag.deletedTriggered when a tag is deleted from a model versionModelVersionTagDeletedPayload
model_version_alias.createdTriggered when an alias is created for a model versionModelVersionAliasCreatedPayload
model_version_alias.deletedTriggered when an alias is deleted from a model versionModelVersionAliasDeletedPayload

快速开始

创建 Webhook

from mlflow import MlflowClient

client = MlflowClient()

# Create a webhook for model version creation events
webhook = client.create_webhook(
name="model-version-notifier",
url="https://your-app.com/webhook",
events=["model_version.created"],
description="Notifies when new model versions are created",
secret="your-secret-key", # Optional: for HMAC signature verification
)

print(f"Created webhook: {webhook.webhook_id}")

测试 Webhook

在将 webhook 推向生产环境之前,使用 MlflowClient.test_webhook() 通过示例有效负载进行测试:

# Test the webhook with an example payload
result = client.test_webhook(webhook.webhook_id)

if result.success:
print(f"Webhook test successful! Status code: {result.response_status}")
else:
print(f"Webhook test failed. Status: {result.response_status}")
if result.error_message:
print(f"Error: {result.error_message}")

您也可以测试特定的事件类型:

# Test with a specific event type
result = client.test_webhook(webhook.webhook_id, event="model_version.created")

当你调用 test_webhook() 时,MLflow 会向你的 webhook URL 发送示例负载。这些测试负载与真实事件负载具有相同的结构。点击上表中的有效负载模式链接以查看每种事件类型的确切结构和示例。

测试多事件 Webhook

如果你的 webhook 订阅了多个事件,test_webhook() 的行为取决于你是否指定了事件:

  • 未指定事件: MLflow 使用 webhook 的事件列表中的 第一个事件
  • 针对特定事件: MLflow 使用指定事件 (必须在 webhook 的事件列表中)
# Create webhook with multiple events
webhook = client.create_webhook(
name="multi-event-webhook",
url="https://your-domain.com/webhook",
events=[
"registered_model.created",
"model_version.created",
"model_version_tag.set",
],
secret="your-secret-key",
)

# Test with first event (registered_model.created)
result = client.test_webhook(webhook.webhook_id)

# Test with specific event
result = client.test_webhook(
webhook.webhook_id,
event=("model_version_tag.set"),
)

Webhook 管理

列出 Webhooks

使用 MlflowClient.list_webhooks() 来检索 webhooks。该方法返回分页结果:

# List webhooks with pagination
webhooks = client.list_webhooks(max_results=10)
for webhook in webhooks:
print(f"{webhook.name}: {webhook.url} (Status: {webhook.status})")
print(f" Events: {', '.join(webhook.events)}")

# Continue to next page if available
if webhooks.next_page_token:
next_page = client.list_webhooks(
max_results=10, page_token=webhooks.next_page_token
)

检索跨多页的所有 webhooks:

# Retrieve all webhooks across pages
all_webhooks = []
page_token = None

while True:
page = client.list_webhooks(max_results=100, page_token=page_token)
all_webhooks.extend(page)

if not page.next_page_token:
break
page_token = page.next_page_token

print(f"Total webhooks: {len(all_webhooks)}")

获取特定的 Webhook

使用 MlflowClient.get_webhook() 检索特定 webhook 的详细信息:

# Get a specific webhook by ID
webhook = client.get_webhook(webhook_id)
print(f"Name: {webhook.name}")
print(f"URL: {webhook.url}")
print(f"Status: {webhook.status}")
print(f"Events: {webhook.events}")

更新 Webhook

使用 MlflowClient.update_webhook() 修改 webhook 配置:

# Update webhook configuration
client.update_webhook(
# Unspecified fields will remain unchanged
webhook_id=webhook.webhook_id,
status="DISABLED", # Temporarily disable the webhook
events=[
"model_version.created",
"model_version_tag.set",
],
)

删除 Webhook

使用 MlflowClient.delete_webhook() 删除 webhook:

# Delete a webhook
client.delete_webhook(webhook.webhook_id)

安全

HMAC 签名验证

当你使用 secret 创建 webhook 时,MLflow 使用 HMAC-SHA256 签名对每个请求进行签名。这允许你的端点验证该请求确实来自 MLflow。签名包含在 X-MLflow-Signature 请求头中,格式为:v1,。参见下方的 FastAPI 示例,了解完整的签名验证实现。

时间戳新鲜度检查

为防止重放攻击,建议验证 webhook 的时间戳是否为近期。X-MLflow-Timestamp 头包含一个 Unix 时间戳,表明 webhook 何时发送。你应拒绝时间戳过旧的 webhook(例如,超过 5 分钟)。

环境变量

  • MLFLOW_WEBHOOK_SECRET_ENCRYPTION_KEY: 用于安全存储 webhook 密钥的加密密钥
  • MLFLOW_WEBHOOK_REQUEST_TIMEOUT: 以秒为单位的 webhook HTTP 请求超时时间(默认:30)
  • MLFLOW_WEBHOOK_REQUEST_MAX_RETRIES: 对失败的 webhook 请求的最大重试次数(默认:3)
  • MLFLOW_WEBHOOK_DELIVERY_MAX_WORKERS: webhook 投递的最大工作线程数(默认:10)

Webhook 有效负载结构

MLflow webhooks 发送具有以下格式的结构化 JSON 有效负载:

{
"entity": "model_version",
"action": "created",
"timestamp": "2025-07-31T08:27:32.080217+00:00",
"data": {
"name": "example_model",
"version": "1",
"source": "models:/123",
"run_id": "abcd1234abcd5678",
"tags": {"example_key": "example_value"},
"description": "An example model version"
}
}

有效载荷字段

  • entity: 触发 webhook 的 MLflow 实体类型(例如,"registered_model""model_version""model_version_tag""model_version_alias"
  • action: 执行的操作(例如 "created""updated""deleted""set"
  • timestamp: 表示 webhook 何时发送的 ISO 8601 时间戳
  • data: 实际的负载数据,包含与实体相关的特定信息(参见上方事件表中的 payload schema 链接)

这种结构化的格式使得以下操作变得容易:

  • 按实体类型或操作筛选 webhooks
  • 使用专用处理程序处理不同的事件类型
  • 在不解析整个有效负载的情况下提取元数据

Webhook 投递可靠性

自动重试逻辑

MLflow 实现了自动重试逻辑以确保 webhook 的可靠投递。当 webhook 请求失败时,MLflow 会对以下状态码自动重试该请求。其他所有状态码不予重试:

状态码类别描述
429速率限制请求过多 - 速率限制错误
500服务器错误内部服务器错误 - 可能是暂时性的服务器错误
502服务器错误错误的网关 - 网关错误
503服务器错误服务不可用 - 服务暂时不可用
504服务器错误网关超时 - 网关超时错误

重试行为

当发生可重试错误时,MLflow:

  1. 指数退避:使用带抖动的指数退避以防止惊群问题

    • 基础延迟:1秒、2秒、4秒、8秒等。
    • 最大退避:上限为 60 秒
    • 抖动:向每个延迟添加最多1秒的随机抖动(需要 urllib3 >= 2.0)
  2. 遵守速率限制:对于 429 响应,MLflow 检查 Retry-After 头并使用以下较大者:

    • Retry-After 头中指定的值
    • 计算出的指数退避时间
  3. 可配置的重试次数: 使用 MLFLOW_WEBHOOK_REQUEST_MAX_RETRIES 环境变量设置最大重试次数

示例:FastAPI Webhook 接收器

下面是一个完整的 FastAPI 应用示例,用于接收和处理 MLflow webhooks:

from fastapi import FastAPI, Request, HTTPException, Header
from typing import Optional
import hmac
import hashlib
import base64
import logging
import time

app = FastAPI()
logger = logging.getLogger(__name__)

# Your webhook secret (keep this secure!)
WEBHOOK_SECRET = "your-secret-key"

# Maximum allowed age for webhook timestamps (in seconds)
MAX_TIMESTAMP_AGE = 300 # 5 minutes


def verify_timestamp_freshness(
timestamp_str: str, max_age: int = MAX_TIMESTAMP_AGE
) -> bool:
"""Verify that the webhook timestamp is recent enough to prevent replay attacks"""
try:
webhook_timestamp = int(timestamp_str)
current_timestamp = int(time.time())
age = current_timestamp - webhook_timestamp
return 0 <= age <= max_age
except (ValueError, TypeError):
return False


def verify_mlflow_signature(
payload: str, signature: str, secret: str, delivery_id: str, timestamp: str
) -> bool:
"""Verify the HMAC signature from MLflow webhook"""
# Extract the base64 signature part (remove 'v1,' prefix)
if not signature.startswith("v1,"):
return False

signature_b64 = signature.removeprefix("v1,")
# Reconstruct the signed content: delivery_id.timestamp.payload
signed_content = f"{delivery_id}.{timestamp}.{payload}"
# Generate expected signature
expected_signature = hmac.new(
secret.encode("utf-8"), signed_content.encode("utf-8"), hashlib.sha256
).digest()
expected_signature_b64 = base64.b64encode(expected_signature).decode("utf-8")
return hmac.compare_digest(signature_b64, expected_signature_b64)


@app.post("/webhook")
async def handle_webhook(
request: Request,
x_mlflow_signature: Optional[str] = Header(None),
x_mlflow_delivery_id: Optional[str] = Header(None),
x_mlflow_timestamp: Optional[str] = Header(None),
):
"""Handle webhook with HMAC signature verification"""

# Get raw payload for signature verification
payload_bytes = await request.body()
payload = payload_bytes.decode("utf-8")

# Verify required headers are present
if not x_mlflow_signature:
raise HTTPException(status_code=400, detail="Missing signature header")
if not x_mlflow_delivery_id:
raise HTTPException(status_code=400, detail="Missing delivery ID header")
if not x_mlflow_timestamp:
raise HTTPException(status_code=400, detail="Missing timestamp header")

# Verify timestamp freshness to prevent replay attacks
if not verify_timestamp_freshness(x_mlflow_timestamp):
raise HTTPException(
status_code=400,
detail="Timestamp is too old or invalid (possible replay attack)",
)

# Verify signature
if not verify_mlflow_signature(
payload,
x_mlflow_signature,
WEBHOOK_SECRET,
x_mlflow_delivery_id,
x_mlflow_timestamp,
):
raise HTTPException(status_code=401, detail="Invalid signature")

# Parse payload
webhook_data = await request.json()

# Extract webhook metadata
entity = webhook_data.get("entity")
action = webhook_data.get("action")
timestamp = webhook_data.get("timestamp")
payload_data = webhook_data.get("data", {})

# Print the payload for debugging
print(f"Received webhook: {entity}.{action}")
print(f"Timestamp: {timestamp}")
print(f"Delivery ID: {x_mlflow_delivery_id}")
print(f"Payload: {payload_data}")

# Add your webhook processing logic here
# For example, handle different event types
if entity == "model_version" and action == "created":
model_name = payload_data.get("name")
version = payload_data.get("version")
print(f"New model version: {model_name} v{version}")
# Add your model version processing logic here
elif entity == "registered_model" and action == "created":
model_name = payload_data.get("name")
print(f"New registered model: {model_name}")
# Add your registered model processing logic here
elif entity == "model_version_tag" and action == "set":
model_name = payload_data.get("name")
version = payload_data.get("version")
tag_key = payload_data.get("key")
tag_value = payload_data.get("value")
print(f"Tag set on {model_name} v{version}: {tag_key}={tag_value}")
# Add your tag processing logic here

return {"status": "success"}


@app.get("/health")
async def health():
"""Health check endpoint"""
return {"status": "healthy"}


if __name__ == "__main__":
import uvicorn

uvicorn.run(app, host="0.0.0.0", port=8000)

运行示例

  1. 安装依赖:

    pip install fastapi uvicorn
  2. 使用 webhook 加密设置 MLflow 服务器:

    # 为 webhook 秘密生成一个安全的加密密钥
    export MLFLOW_WEBHOOK_SECRET_ENCRYPTION_KEY=$(python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())")

    # 启动支持 webhook 的 MLflow 服务器
    mlflow server --backend-store-uri sqlite:///mlflow.db
  3. 启动 webhook 接收器:

    python webhook_receiver.py
  4. 配置 MLflow webhook:

    from mlflow import MlflowClient

    client = MlflowClient("http://localhost:5000")

    # 使用 HMAC 验证创建 webhook
    webhook = client.create_webhook(
    name="fastapi-receiver",
    url="https://your-domain.com/webhook",
    events=["model_version.created"],
    secret="your-secret-key",
    )
  5. 测试 webhook:

    # 测试 webhook 连通性
    result = client.test_webhook(webhook.webhook_id)
    print(f"Test result: {result.success}")

    # 创建模型版本以触发 webhook
    client.create_registered_model("test-model")
    client.create_model_version(
    name="test-model", source="s3://bucket/model", run_id="abc123"
    )

故障排除

常见问题

  1. Webhook 未触发:

    • 验证 webhook 状态为 "ACTIVE"
    • 检查事件类型是否与您的操作匹配
    • 确保您的 MLflow 服务器可以访问 webhook URL
  2. 签名验证失败:

    • 确保您正在使用原始请求体进行验证
    • 检查密钥是否完全匹配(没有多余的空格)
  3. 连接超时:

    • MLflow 对 webhook 请求的默认超时为 30 秒(可通过 MLFLOW_WEBHOOK_REQUEST_TIMEOUT 配置)
    • 确保你的端点快速响应,或在需要时增加超时时间

API 参考

有关完整的 API 文档,请参见: