使用LLMs进行间接推理

背景

Zhang等人（2024）最近提出了一种间接推理方法，以增强LLMs的推理能力。该方法利用反面和矛盾的逻辑来处理IR任务，如事实推理和数学证明。它包括两个关键步骤：1）通过增加数据和规则（即反命题的逻辑等价）来增强LLMs的可理解性，以及2）设计提示模板以激发LLMs根据反证法进行间接推理。

在LLMs（如GPT-3.5-turbo和Gemini-pro）上的实验表明，与传统的直接推理方法相比，所提出的方法使事实推理的整体准确性提高了27.33%，数学证明的准确性提高了31.43%。

以下是零样本反证法模板的一个示例。

提示

如果 a+|a|=0，试着证明 a<0。

步骤1：列出原命题中的条件和问题。

步骤2：将步骤1中列出的条件合并为一个。定义为 wj。

步骤3：让我们一步一步地思考。请考虑所有可能性。如果wj（步骤2中定义的）和问题的否定在至少一种可能性中不为空，则原命题为假。否则，原命题为真。

答案：

代码 / API

GPT-4 (OpenAI)

from openai import OpenAI
client = OpenAI()

response = client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[
{
    "role": "user",
    "content": "If a+|a|=0, try to prove that a<0.\n\nStep 1: List the conditions and questions in the original proposition.\n\nStep 2: Merge the conditions listed in Step 1 into one. Define it as wj.\n\nStep 3: Let us think it step by step. Please consider all possibilities. If the intersection between wj (defined in Step 2) and the negation of the question is not empty at least in one possibility, the original proposition is false. Otherwise, the original proposition is true.\n\nAnswer:"
}
],
temperature=0,
max_tokens=1000,
top_p=1,
frequency_penalty=0,
presence_penalty=0
)

Mixtral MoE 8x7B Instruct (Fireworks)

import fireworks.client
fireworks.client.api_key = "<FIREWORKS_API_KEY>"
completion = fireworks.client.ChatCompletion.create(
   model="accounts/fireworks/models/mixtral-8x7b-instruct",
   messages=[
       {
       "role": "user",
       "content": "If a+|a|=0, try to prove that a<0.\n\nStep 1: List the conditions and questions in the original proposition.\n\nStep 2: Merge the conditions listed in Step 1 into one. Define it as wj.\n\nStep 3: Let us think it step by step. Please consider all possibilities. If the intersection between wj (defined in Step 2) and the negation of the question is not empty at least in one possibility, the original proposition is false. Otherwise, the original proposition is true.\n\nAnswer:",
       }
   ],
   stop=["<|im_start|>","<|im_end|>","<|endoftext|>"],
   stream=True,
   n=1,
   top_p=1,
   top_k=40,
   presence_penalty=0,
   frequency_penalty=0,
   prompt_truncate_len=1024,
   context_length_exceeded_behavior="truncate",
   temperature=0.9,
   max_tokens=4000
)

参考

• Large Language Models as an Indirect Reasoner: Contrapositive and Contradiction for Automated Reasoning (2024年2月6日)