提取论文中的模型名称

背景

以下提示测试了LLM执行信息提取任务的能力，该任务涉及从机器学习论文摘要中提取模型名称。

提示

你的任务是从机器学习论文摘要中提取模型名称。你的回复是格式为["model_name"]的模型名称数组。如果你在摘要中没有找到模型名称或者不确定，请返回["NA"]

摘要：大型语言模型（LLMs），如ChatGPT和GPT-4，已经革新了自然语言处理研究，并展示了在通用人工智能（AGI）中的潜力。然而，LLMs的昂贵训练和部署对透明和开放的学术研究提出了挑战。为了解决这些问题，本项目开源了中文LLaMA和Alpaca…

提示模板

你的任务是从机器学习论文摘要中提取模型名称。你的回复是格式为["model_name"]的模型名称数组。如果你在摘要中没有找到模型名称或者不确定，请返回["NA"]

摘要：{input}

代码 / API

GPT-4 (OpenAI)

from openai import OpenAI
client = OpenAI()

response = client.chat.completions.create(
model="gpt-4",
messages=[
    {
    "role": "user",
    "content": "Your task is to extract model names from machine learning paper abstracts. Your response is an array of the model names in the format [\\\"model_name\\\"]. If you don't find model names in the abstract or you are not sure, return [\\\"NA\\\"]\n\nAbstract: Large Language Models (LLMs), such as ChatGPT and GPT-4, have revolutionized natural language processing research and demonstrated potential in Artificial General Intelligence (AGI). However, the expensive training and deployment of LLMs present challenges to transparent and open academic research. To address these issues, this project open-sources the Chinese LLaMA and Alpaca…"
    }
],
temperature=1,
max_tokens=250,
top_p=1,
frequency_penalty=0,
presence_penalty=0
)

Mixtral MoE 8x7B Instruct (Fireworks)

import fireworks.client
fireworks.client.api_key = "<FIREWORKS_API_KEY>"
completion = fireworks.client.ChatCompletion.create(
    model="accounts/fireworks/models/mixtral-8x7b-instruct",
    messages=[
        {
        "role": "user",
        "content": "Your task is to extract model names from machine learning paper abstracts. Your response is an array of the model names in the format [\\\"model_name\\\"]. If you don't find model names in the abstract or you are not sure, return [\\\"NA\\\"]\n\nAbstract: Large Language Models (LLMs), such as ChatGPT and GPT-4, have revolutionized natural language processing research and demonstrated potential in Artificial General Intelligence (AGI). However, the expensive training and deployment of LLMs present challenges to transparent and open academic research. To address these issues, this project open-sources the Chinese LLaMA and Alpaca…",
        }
    ],
    stop=["<|im_start|>","<|im_end|>","<|endoftext|>"],
    stream=True,
    n=1,
    top_p=1,
    top_k=40,
    presence_penalty=0,
    frequency_penalty=0,
    prompt_truncate_len=1024,
    context_length_exceeded_behavior="truncate",
    temperature=0.9,
    max_tokens=4000
)

参考

• Prompt Engineering Guide (2023年3月16日)