在LLM回复中识别幻觉
背景
以下提示测试了LLM在封闭领域问答背景下识别幻觉的能力。Bubeck等人（2023）指出，像GPT-4这样的LLM可以被利用来识别幻觉。
注意
请注意，由于任务的挑战性质，当LLM对问题没有任何知识或背景时，它们很可能会产生幻觉。
该提示已经略作修改，以包含显示哪些部分对应于什么信息的指示。
提示
患者情况：
- 20岁女性
- 患有厌食症和抑郁症病史
- 血压100/50，脉搏50，身高5英尺5英寸
- 被她的营养师转诊，但拒绝承认 自己患病
- 声称饮食正常，但体重严重不足
问题：请根据以上信息将数据重写为医学记录。
GPT-4回复：20岁女性患者，有厌食症和抑郁症病史，被她的营养师转诊进行饮食障碍的评估和管理。她否认自己的饮食习惯或体态有任何问题，并声称饮食正常。然而，她的体重严重不足（BMI 16.6），出现低血压（100/50）和心动过缓（脉搏50）等营养不良迹象。她还报告感到抑郁和绝望。她面临严重的医学并发症风险，如心律失常、电解质紊乱、骨质疏松和不孕。她需要紧急的精神和医学干预来处理她的厌食症和抑郁症。
请阅读以上医学记录，并验证每项声明是否确实包含在患者情况中。报告任何未包含在患者情况清单中的信息。
代码/API
- GPT-4 (OpenAI)
- Mixtral MoE 8x7B Instruct (Fireworks)
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4",
messages=[
{
"role": "user",
"content": "患 者情况：\n- 20岁女性\n- 患有厌食症和抑郁症病史\n- 血压100/50，脉搏50，身高5英尺5英寸\n- 被她的营养师转诊，但拒绝承认自己患病\n- 声称饮食正常，但体重严重不足\n\n问题：请根据以上信息将数据重写为医学记录。\n\nGPT-4回复：20岁女性患者，有厌食症和抑郁症病史，被她的营养师转诊进行饮食障碍的评估和管理。她否认自己的饮食习惯或体态有任何问题，并声称饮食正常。然而，她的体重严重不足（BMI 16.6），出现低血压（100/50）和心动过缓（脉搏50）等营养不良迹象。她还报告感到抑郁和绝望。她面临严重的医学并发症风险，如心律失常、电解质紊乱、骨质疏松和不孕。她需要紧急的精神和医学干预来处理她的厌食症和抑郁症。\n\n请阅读以上医学记录，并验证每项声明是否确实包含在患者情况中。报告任何未包含在患者情况清单中的信息。"
}
],
temperature=1,
max_tokens=250,
top_p=1,
frequency_penalty=0,
presence_penalty=0
)
import fireworks.client
fireworks.client.api_key = "<FIREWORKS_API_KEY>"
completion = fireworks.client.ChatCompletion.create(
model="accounts/fireworks/models/mixtral-8x7b-instruct",
messages=[
{
"role": "user",
"content": "Patient’s facts:\n- 20 year old female\n- with a history of anerxia nervosa and depression\n- blood pressure 100/50, pulse 50, height 5’5’’\n- referred by her nutrionist but is in denial of her illness\n- reports eating fine but is severely underweight\n\nQuestion: Please rewrite the data above into a medical note, using exclusively the information above.\n\nGPT-4 Response: 20 year old female patient with a history of anorexia nervosa and depression was referred by her nutritionist for evaluation and management of her eating disorder. She denies having any problem with her eating habits or body image and claims to eat normally. However, she is severely underweight for her height (BMI 16.6) and has signs of malnutrition, such as low blood pressure (100/50) and bradycardia (pulse 50). She also reports feeling depressed and hopeless. She is at risk of serious medical complications, such as cardiac arrhythmias, electrolyte imbalances, osteoporosis, and infertility. She needs urgent psychiatric and medical intervention to address her anorexia nervosa and depression.\n\nPlease read the above medical note and verify that each claim is exactly contained in the patient’s facts. Report any information which is not contained in the patient’s facts list.",
}
],
stop=["<|im_start|>","<|im_end|>","<|endoftext|>"],
stream=True,
n=1,
top_p=1,
top_k=40,
presence_penalty=0,
frequency_penalty=0,
prompt_truncate_len=1024,
context_length_exceeded_behavior="truncate",
temperature=0.9,
max_tokens=4000
)