GPT-3.5-Turbo 微调¶

在这个笔记本中，我们将通过一个 GPT-3.5-Turbo 微调的示例来介绍。

具体来说，我们尝试通过使用 GPT-4 生成训练数据，然后用这些数据对 GPT-3.5 进行微调，来提炼 GPT-4 的知识。

所有训练数据都是使用我们索引数据的两个不同部分生成的，分别创建了训练集和评估集。

然后我们使用我们的 OpenAIFinetuneEngine 包装器进行微调。

评估是使用 ragas 库进行的，我们稍后会详细介绍。

In [ ]:

%pip install llama-index-finetuning
%pip install llama-index-finetuning-callbacks
%pip install llama-index-llms-openai

%pip install llama-index-finetuning %pip install llama-index-finetuning-callbacks %pip install llama-index-llms-openai

In [ ]:

# !pip install llama-index pypdf sentence-transformers ragas

# !pip install llama-index pypdf sentence-transformers ragas

In [ ]:

import os
import openai

import os import openai

In [ ]:

os.environ["OPENAI_API_KEY"] = "sk-..."
openai.api_key = os.environ["OPENAI_API_KEY"]

os.environ["OPENAI_API_KEY"] = "sk-..." openai.api_key = os.environ["OPENAI_API_KEY"]

数据设置¶

在这里，我们首先下载PDF文件，这将用于生成训练数据。

In [ ]:

!curl https://www.ipcc.ch/report/ar6/wg2/downloads/report/IPCC_AR6_WGII_Chapter03.pdf --output IPCC_AR6_WGII_Chapter03.pdf

!curl https://www.ipcc.ch/report/ar6/wg2/downloads/report/IPCC_AR6_WGII_Chapter03.pdf --output IPCC_AR6_WGII_Chapter03.pdf

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 20.7M  100 20.7M    0     0   397k      0  0:00:53  0:00:53 --:--:--  417k84k      0  0:00:55  0:00:24  0:00:31  406k    0   395k      0  0:00:53  0:00:48  0:00:05  403k0   396k      0  0:00:53  0:00:53 --:--:--  406k

下一步是生成训练和评估数据集。

我们将在我们下载的PDF的不同部分生成40个问题。

我们可以使用GPT-3.5来对评估问题进行处理，以获得基准性能。

然后，我们将使用GPT-4来处理训练问题，生成我们的训练数据。训练数据将使用我们的OpenAIFineTuningHandler进行收集。

如果您不想花费时间/代币，这一步是完全可选的--评估和训练问题也已经在这个文件夹中提供，以及训练数据！

训练生成¶

这个notebook包含了一个用于生成训练数据的示例。我们将使用Python中的一些流行的库来生成数据，并将其用于训练模型。

In [ ]:

from llama_index.core import SimpleDirectoryReaderfrom llama_index.llms.openai import OpenAIfrom llama_index.core.evaluation import DatasetGeneratordocuments = SimpleDirectoryReader(    input_files=["IPCC_AR6_WGII_Chapter03.pdf"]).load_data()# 打乱文档顺序import randomrandom.seed(42)random.shuffle(documents)gpt_35_llm = OpenAI(model="gpt-3.5-turbo", temperature=0.3)

from llama_index.core import SimpleDirectoryReaderfrom llama_index.llms.openai import OpenAIfrom llama_index.core.evaluation import DatasetGeneratordocuments = SimpleDirectoryReader( input_files=["IPCC_AR6_WGII_Chapter03.pdf"]).load_data()# 打乱文档顺序import randomrandom.seed(42)random.shuffle(documents)gpt_35_llm = OpenAI(model="gpt-3.5-turbo", temperature=0.3)

In [ ]:

question_gen_query = (
    "You are a Teacher/ Professor. Your task is to setup "
    "a quiz/examination. Using the provided context, formulate "
    "a single question that captures an important fact from the "
    "context. Restrict the question to the context information provided."
)

dataset_generator = DatasetGenerator.from_documents(
    documents[:50],
    question_gen_query=question_gen_query,
    llm=gpt_35_llm,
)

question_gen_query = ( "You are a Teacher/ Professor. Your task is to setup " "a quiz/examination. Using the provided context, formulate " "a single question that captures an important fact from the " "context. Restrict the question to the context information provided." ) dataset_generator = DatasetGenerator.from_documents( documents[:50], question_gen_query=question_gen_query, llm=gpt_35_llm, )

In [ ]:

# 注意：这可能需要一些时间。去喝杯咖啡吧！questions = dataset_generator.generate_questions_from_nodes(num=40)print("生成了 ", len(questions), " 个问题")

# 注意：这可能需要一些时间。去喝杯咖啡吧！questions = dataset_generator.generate_questions_from_nodes(num=40)print("生成了 ", len(questions), " 个问题")

Generated  40  questions

In [ ]:

with open("train_questions.txt", "w") as f:
    for question in questions:
        f.write(question + "\n")

with open("train_questions.txt", "w") as f: for question in questions: f.write(question + "\n")

评估生成¶

现在，让我们在一组完全不同的文档上生成问题，以创建我们的评估数据集。

In [ ]:

dataset_generator = DatasetGenerator.from_documents(    documents[        50:    ],  # 由于我们为40个文档生成了大约1个问题，因此我们可以跳过前40个    question_gen_query=question_gen_query,    llm=gpt_35_llm,)

dataset_generator = DatasetGenerator.from_documents( documents[ 50: ], # 由于我们为40个文档生成了大约1个问题，因此我们可以跳过前40个 question_gen_query=question_gen_query, llm=gpt_35_llm,)

In [ ]:

# 注意：这可能需要一些时间。去喝杯咖啡吧！questions = dataset_generator.generate_questions_from_nodes(num=40)print("生成了 ", len(questions), " 个问题")

# 注意：这可能需要一些时间。去喝杯咖啡吧！questions = dataset_generator.generate_questions_from_nodes(num=40)print("生成了 ", len(questions), " 个问题")

Generated  40  questions

In [ ]:

with open("eval_questions.txt", "w") as f:
    for question in questions:
        f.write(question + "\n")

with open("eval_questions.txt", "w") as f: for question in questions: f.write(question + "\n")

使用 GPT-3.5-Turbo 查询引擎进行初始评估¶

在这个评估中，我们将使用 ragas 评估库。

Ragas 包含大量用于 RAG 管道的评估指标，您可以在这里阅读相关内容。

在这个笔记本中，我们将使用以下两个指标：

• answer_relevancy - 这个指标衡量生成的答案与提示的相关程度。如果生成的答案不完整或包含冗余信息，则得分会较低。这是通过计算使用生成的答案生成给定问题的可能性来量化的。取值范围为 (0,1)，值越高越好。
• faithfulness - 这个指标衡量生成的答案在给定上下文中的事实一致性。这是通过一个多步骤范式来实现的，包括从生成的答案创建语句，然后对每个语句针对上下文进行验证。答案的范围为 (0,1)，值越高越好。

In [ ]:

questions = []
with open("eval_questions.txt", "r") as f:
    for line in f:
        questions.append(line.strip())

questions = [] with open("eval_questions.txt", "r") as f: for line in f: questions.append(line.strip())

In [ ]:

from llama_index.core import VectorStoreIndex# 将上下文窗口限制为2048个标记，以便使用refinefrom llama_index.core import SettingsSettings.context_window = 2048index = VectorStoreIndex.from_documents(    documents,)query_engine = index.as_query_engine(similarity_top_k=2, llm=gpt_35_llm)

from llama_index.core import VectorStoreIndex# 将上下文窗口限制为2048个标记，以便使用refinefrom llama_index.core import SettingsSettings.context_window = 2048index = VectorStoreIndex.from_documents( documents,)query_engine = index.as_query_engine(similarity_top_k=2, llm=gpt_35_llm)

In [ ]:

contexts = []
answers = []

for question in questions:
    response = query_engine.query(question)
    contexts.append([x.node.get_content() for x in response.source_nodes])
    answers.append(str(response))

contexts = [] answers = [] for question in questions: response = query_engine.query(question) contexts.append([x.node.get_content() for x in response.source_nodes]) answers.append(str(response))

In [ ]:

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import answer_relevancy, faithfulness

ds = Dataset.from_dict(
    {
        "question": questions,
        "answer": answers,
        "contexts": contexts,
    }
)

result = evaluate(ds, [answer_relevancy, faithfulness])
print(result)

from datasets import Dataset from ragas import evaluate from ragas.metrics import answer_relevancy, faithfulness ds = Dataset.from_dict( { "question": questions, "answer": answers, "contexts": contexts, } ) result = evaluate(ds, [answer_relevancy, faithfulness]) print(result)

evaluating with [answer_relevancy]

100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [01:02<00:00, 20.69s/it]

evaluating with [faithfulness]

100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [03:52<00:00, 77.37s/it]

{'ragas_score': 0.8356, 'answer_relevancy': 0.9725, 'faithfulness': 0.7325}

使用GPT-4收集训练数据¶

在这里，我们使用GPT-4和OpenAIFineTuningHandler来收集我们想要训练的数据。

In [ ]:

from llama_index.llms.openai import OpenAI
from llama_index.finetuning.callbacks import OpenAIFineTuningHandler
from llama_index.core.callbacks import CallbackManager

finetuning_handler = OpenAIFineTuningHandler()
callback_manager = CallbackManager([finetuning_handler])

llm = OpenAI(model="gpt-3.5-turbo", temperature=0.3)
llm.callback_manager = callback_manager

from llama_index.llms.openai import OpenAI from llama_index.finetuning.callbacks import OpenAIFineTuningHandler from llama_index.core.callbacks import CallbackManager finetuning_handler = OpenAIFineTuningHandler() callback_manager = CallbackManager([finetuning_handler]) llm = OpenAI(model="gpt-3.5-turbo", temperature=0.3) llm.callback_manager = callback_manager

In [ ]:

questions = []
with open("train_questions.txt", "r") as f:
    for line in f:
        questions.append(line.strip())

questions = [] with open("train_questions.txt", "r") as f: for line in f: questions.append(line.strip())

In [ ]:

from llama_index.core import VectorStoreIndex

index = VectorStoreIndex.from_documents(
    documents,
)

query_engine = index.as_query_engine(similarity_top_k=2, llm=llm)

from llama_index.core import VectorStoreIndex index = VectorStoreIndex.from_documents( documents, ) query_engine = index.as_query_engine(similarity_top_k=2, llm=llm)

In [ ]:

for question in questions:
    response = query_engine.query(question)

for question in questions: response = query_engine.query(question)

创建OpenAIFinetuneEngine¶

我们创建一个OpenAIFinetuneEngine：这个微调引擎将负责启动微调作业，并返回一个LLM模型，您可以直接将其插入到LlamaIndex工作流的其余部分中。

我们使用默认构造函数，但我们也可以直接使用from_finetuning_handler类方法将我们的微调处理程序传递给这个引擎。

In [ ]:

finetuning_handler.save_finetuning_events("finetuning_events.jsonl")

finetuning_handler.save_finetuning_events("finetuning_events.jsonl")

In [ ]:

from llama_index.finetuning import OpenAIFinetuneEnginefinetune_engine = OpenAIFinetuneEngine(    "gpt-3.5-turbo",    "finetuning_events.jsonl",    # start_job_id="<start-job-id>"  # 如果您有现有的作业，可以在这里指定id)# finetune_engine = OpenAIFinetuneEngine.from_finetuning_handler(#     finetuning_handler,#     "gpt-3.5-turbo",#     "tmp.jsonl"# )

from llama_index.finetuning import OpenAIFinetuneEnginefinetune_engine = OpenAIFinetuneEngine( "gpt-3.5-turbo", "finetuning_events.jsonl", # start_job_id="" # 如果您有现有的作业，可以在这里指定id)# finetune_engine = OpenAIFinetuneEngine.from_finetuning_handler(# finetuning_handler,# "gpt-3.5-turbo",# "tmp.jsonl"# )

In [ ]:

finetune_engine.finetune()

finetune_engine.finetune()

Num examples: 61
First example:
{'role': 'system', 'content': "You are an expert Q&A system that is trusted around the world.\nAlways answer the query using the provided context information, and not prior knowledge.\nSome rules to follow:\n1. Never directly reference the given context in your answer.\n2. Avoid statements like 'Based on the context, ...' or 'The context information ...' or anything along those lines."}
{'role': 'user', 'content': 'Context information is below.\n---------------------\npage_label: 410\nfile_name: IPCC_AR6_WGII_Chapter03.pdf\n\nIt is challenging to apply this experimental approach to communities or ecosystems (see Figure \nBox\xa03.1.1).To date, most research on community or ecosystem response to climate-induced drivers has been in large-volume (>10,000 l) \nmesocosms (Riebesell and Gattuso, 2014), or at natural analogues such as CO 2 seeps, in which only one driver (ocean acidification) is \naltered (see (4) in Figure Box\xa03.1.1).Only very recently have two drivers been incorporated into climate-change manipulation studies \nexamining responses of primary producers to secondary consumers (see (5) in Figure Box\xa03.1.1a; Nagelkerken et\xa0al., 2020).Therefore, \n‘natural experiments’ from the geological past (Reddin et\xa0al., 2020) provide insights into how food webs and their constituents respond to \ncomplex change involving multiple drivers.Contemporary observations are occasionally long enough (>50\xa0years) to capture community \nresponses to complex climate change.For example, Brun et\xa0al.(2019) reported a shift in zooplankton community structure in the North \nAtlantic (1960–2014), with major biogeochemical ramifications.Conducting sufficiently long manipulation experiments to study the effect of adaptation on organisms is equally difficult (see Figure \nBox\xa03.1.1b), with much research restricted to multi-year studies of the microevolution of fast-growing (more than one division per day) \nphytoplankton species responding to single drivers (Lohbeck et\xa0al., 2012; Schaum et\xa0al., 2016).In a few experimental evolution studies \n(see (7) in Figure Box\xa03.1.1a; Brennan et\xa0al., 2017), multiple drivers have been used, but none have used communities or ecosystems (see \nFigure Box\xa03.1.1b).Nevertheless, the fossil record provides limited evidence of adaptations to less rapid (relative to present day) climate \nchange (Jackson et\xa0al., 2018).Despite the need to explore ecological or biogeochemical responses to projected future ocean conditions, \nlogistical challenges require that assessments of climate-change impacts at scales larger than mesocosms use large-scale, long-term in \nsitu observational studies (as documented in Section\xa03.4).\n\npage_label: 409\nfile_name: IPCC_AR6_WGII_Chapter03.pdf\n\n3\n409Oceans and Coastal Ecosystems and Their Services  Chapter 3\nunderlies inhibited thermal adaptation under nitrogen-limited \nconditions (low confidence) (Aranguren-Gassis et\xa0 al., 2019).When \nselection is strong due to unfavourable environmental conditions, \nmicrobial populations can encounter functional and evolutionary \ntrade-offs evidenced by reducing growth rates while increasing \ntolerance and metabolism of reactive oxygen species (Lindberg and \nCollins, 2020).Other trade-offs can be observed in offspring quality \nand number (Lindberg and Collins, 2020).These findings contribute \ntowards a mechanistic framework describing the range of evolutionary \nstrategies in response to multiple drivers (Collins et\xa0al., 2020), but other \nhazards, such as extreme events (e.g., MHWs), still need to be included \nbecause their characteristics may alter the potential for adaptation of \nspecies and populations to climate change (Gruber et\xa0al., 2021).3.3.5 Ecological Response to Multiple Drivers\nAssessing ecological responses to multiple climate-induced drivers \nrequires a combination of approaches, including laboratory- and \nfield-based experiments, field observations (e.g., natural gradients, \nclimate analogues), study of paleo-analogues and the development \nof mechanistic and empirical models (Clapham, 2019; Gissi et\xa0 al., \n2021).Experimental studies of food-web responses are often limited \nto an individual driver, although recent manipulations have used a \nmatrix of >1000-l mesocosms to explore ecological responses to both \nwarming and acidification (see Box\xa0 3.1; Nagelkerken et\xa0 al., 2020).Hence, complementary approaches are needed to indirectly explore \nthe mechanisms underlying ecosystem responses to global climate \nchange (Parmesan et\xa0al., 2013).Observations from time series longer \nthan modes of natural variability (i.e., decades) are essential for \nrevealing and attributing ecological responses to climate change (e.g., \nSection\xa03.4; Barton et\xa0al., 2015b; Brun et\xa0al., 2019).Also, paleorecords \nprovide insights into the influence of multiple drivers on marine \nbiota (Cross-Chapter Box\xa0 PALEO in Chapter\xa0 1; Reddin et\xa0 al., 2020).Specifically, associations between vulnerabilities and traits of marine \nectotherms in laboratory experiments correspond with organismal \nresponses to ancient hyperthermal events (medium confidence) \n(Reddin et\xa0 al., 2020).This corroboration suggests that responses to \nmultiple drivers inferred from the fossil record can help provide insights \ninto the future status of functional groups, and hence food webs, under \nrapid climate change.Multi-species and integrated end-to-end ecosystem models are \npowerful tools to explore and project outcomes to the often-interacting \ncumulative effects of climate change and other anthropogenic drivers \n(Section\xa03.1; Kaplan and Marshall, 2016; Koenigstein et\xa0al., 2016; Peck \nand Pinnegar, 2018; Tittensor et\xa0 al., 2018; Gissi et\xa0 al., 2021).These \nmodels can integrate some aspects of the knowledge accrued from \nmanipulation experiments, paleo- and contemporary observations, help \ntest the relative importance of specific drivers and driver combinations, \nand identify synergistic or antagonistic responses (Koenigstein et\xa0al., \n2016; Payne et\xa0al., 2016; Skogen et\xa0al., 2018; Tittensor et\xa0al., 2018).As these models are associated with wide-ranging uncertainties \n(SM3.2.2; Payne et\xa0 al., 2016; Trolle et\xa0 al., 2019; Heneghan et\xa0 al., \n2021), they cannot be expected to accurately project the trajectories \nof complex marine ecosystems under climate change; hence, they are \nmost useful for assessing overall trends and in particular for providing a plausible envelope of trajectories across a range of assumptions \n(Fulton et\xa0al., 2018; Peck et\xa0al., 2018; Tittensor et\xa0al., 2018).\n---------------------\nGiven the context information and not prior knowledge, answer the query.\nQuery: What are some approaches used to assess ecological responses to multiple climate-induced drivers in the context of climate change and the oceans?\nAnswer: '}
{'role': 'assistant', 'content': 'Several approaches are used to assess ecological responses to multiple climate-induced drivers. These include laboratory- and field-based experiments, field observations such as natural gradients and climate analogues, the study of paleo-analogues, and the development of mechanistic and empirical models. Experimental studies often focus on individual drivers, but recent manipulations have used large-volume mesocosms to explore ecological responses to both warming and acidification. Observations from time series longer than modes of natural variability are essential for revealing and attributing ecological responses to climate change. Paleorecords also provide insights into the influence of multiple drivers on marine biota. Multi-species and integrated end-to-end ecosystem models are powerful tools to explore and project outcomes to the often-interacting cumulative effects of climate change and other anthropogenic drivers. These models can integrate some aspects of the knowledge accrued from manipulation experiments, paleo- and contemporary observations, help test the relative importance of specific drivers and driver combinations, and identify synergistic or antagonistic responses.'}
No errors found
Num examples missing system message: 21
Num examples missing user message: 0

#### Distribution of num_messages_per_example:
min / max: 2, 3
mean / median: 2.6557377049180326, 3.0
p5 / p95: 2.0, 3.0

#### Distribution of num_total_tokens_per_example:
min / max: 229, 2011
mean / median: 1274.27868852459, 1385.0
p5 / p95: 533.0, 1848.0

#### Distribution of num_assistant_tokens_per_example:
min / max: 11, 334
mean / median: 72.36065573770492, 37.0
p5 / p95: 23.0, 193.0

0 examples may be over the 4096 token limit, they will be truncated during fine-tuning
Dataset has ~77731 tokens that will be charged for during training
By default, you'll train for 3 epochs on this dataset
By default, you'll be charged for ~233193 tokens
As of Augest 22, 2023, fine-tuning gpt-3.5-turbo is $0.008 / 1K Tokens.
This means your total cost for training will be $0.621848 per epoch.
Waiting for file to be ready...

In [ ]:

finetune_engine.get_current_job()

finetune_engine.get_current_job()

Out[ ]:

<FineTuningJob fine_tuning.job id=ftjob-u9T7BF5zRxVX4n5b9Jtbb5cR at 0x2c641fe20> JSON: {
  "object": "fine_tuning.job",
  "id": "ftjob-u9T7BF5zRxVX4n5b9Jtbb5cR",
  "model": "gpt-3.5-turbo-0613",
  "created_at": 1693254044,
  "finished_at": null,
  "fine_tuned_model": null,
  "organization_id": "org-1ZDAvajC6v2ZtAP9hLEIsXRz",
  "result_files": [],
  "status": "running",
  "validation_file": null,
  "training_file": "file-j1fwmqIAoqZXWZQ8EqwHucXs",
  "hyperparameters": {
    "n_epochs": 3
  },
  "trained_tokens": null
}

In [ ]:

ft_llm = finetune_engine.get_finetuned_model(temperature=0.3)

ft_llm = finetune_engine.get_finetuned_model(temperature=0.3)

评估¶

经过一段时间，您的模型将完成训练！

接下来的步骤是再次在评估数据集上运行我们经过微调的模型，以衡量任何性能提升。

In [ ]:

from llama_index.llms.openai import OpenAIfrom llama_index.finetuning.callbacks import OpenAIFineTuningHandlerfrom llama_index.core.callbacks import CallbackManager# 选项1：直接将ft_llm传递给Settingsfrom llama_index.core import SettingsSettings.llm = ft_llmSettings.context_window = (    2048  # 人为地限制上下文窗口以测试改进过程)# # 选项2：您也可以手动指定模型名称# ft_model_name = "ft:gpt-3.5-turbo-0613:..."# Settings.llm = OpenAI(model=ft_model_name, temperature=0.3)

from llama_index.llms.openai import OpenAIfrom llama_index.finetuning.callbacks import OpenAIFineTuningHandlerfrom llama_index.core.callbacks import CallbackManager# 选项1：直接将ft_llm传递给Settingsfrom llama_index.core import SettingsSettings.llm = ft_llmSettings.context_window = ( 2048 # 人为地限制上下文窗口以测试改进过程)# # 选项2：您也可以手动指定模型名称# ft_model_name = "ft:gpt-3.5-turbo-0613:..."# Settings.llm = OpenAI(model=ft_model_name, temperature=0.3)

In [ ]:

questions = []
with open("eval_questions.txt", "r") as f:
    for line in f:
        questions.append(line.strip())

questions = [] with open("eval_questions.txt", "r") as f: for line in f: questions.append(line.strip())

In [ ]:

from llama_index.core import VectorStoreIndex

index = VectorStoreIndex.from_documents(documents)

query_engine = index.as_query_engine(similarity_top_k=2, llm=ft_llm)

from llama_index.core import VectorStoreIndex index = VectorStoreIndex.from_documents(documents) query_engine = index.as_query_engine(similarity_top_k=2, llm=ft_llm)

In [ ]:

contexts = []
answers = []

for question in questions:
    response = query_engine.query(question)
    contexts.append([x.node.get_content() for x in response.source_nodes])
    answers.append(str(response))

contexts = [] answers = [] for question in questions: response = query_engine.query(question) contexts.append([x.node.get_content() for x in response.source_nodes]) answers.append(str(response))

In [ ]:

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import answer_relevancy, faithfulness

ds = Dataset.from_dict(
    {
        "question": questions,
        "answer": answers,
        "contexts": contexts,
    }
)

result = evaluate(ds, [answer_relevancy, faithfulness])
print(result)

from datasets import Dataset from ragas import evaluate from ragas.metrics import answer_relevancy, faithfulness ds = Dataset.from_dict( { "question": questions, "answer": answers, "contexts": contexts, } ) result = evaluate(ds, [answer_relevancy, faithfulness]) print(result)

evaluating with [answer_relevancy]

100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:49<00:00, 16.34s/it]

evaluating with [faithfulness]

100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [04:04<00:00, 81.44s/it]

{'ragas_score': 0.8680, 'answer_relevancy': 0.9607, 'faithfulness': 0.7917}

探索差异¶

让我们快速比较一下响应的差异，以证明微调确实改变了一些东西。

In [ ]:

from llama_index.core import VectorStoreIndex

index = VectorStoreIndex.from_documents(documents)

from llama_index.core import VectorStoreIndex index = VectorStoreIndex.from_documents(documents)

In [ ]:

questions = []
with open("eval_questions.txt", "r") as f:
    for line in f:
        questions.append(line.strip())

questions = [] with open("eval_questions.txt", "r") as f: for line in f: questions.append(line.strip())

In [ ]:

print(questions[12])

print(questions[12])

What is a key barrier globally for ocean health, governance, and adaptation to climate change, according to the report?

翻译后¶

原始¶

In [ ]:

from llama_index.core.response.notebook_utils import display_response
from llama_index.llms.openai import OpenAI


gpt_35_llm = OpenAI(model="gpt-3.5-turbo", temperature=0.3)

from llama_index.core.response.notebook_utils import display_response from llama_index.llms.openai import OpenAI gpt_35_llm = OpenAI(model="gpt-3.5-turbo", temperature=0.3)

In [ ]:

query_engine = index.as_query_engine(llm=gpt_35_llm)

response = query_engine.query(questions[12])

display_response(response)

query_engine = index.as_query_engine(llm=gpt_35_llm) response = query_engine.query(questions[12]) display_response(response)

Final Response: A key barrier globally for ocean health, governance, and adaptation to climate change, according to the report, is the availability of technology, knowledge, and financial support, as well as existing governance structures.

微调¶

In [ ]:

query_engine = index.as_query_engine(llm=ft_llm)

response = query_engine.query(questions[12])

display_response(response)

query_engine = index.as_query_engine(llm=ft_llm) response = query_engine.query(questions[12]) display_response(response)

Final Response: The report identifies a broad range of barriers and limits for adaptation to climate change in ecosystems and human systems. These include the availability of technology, knowledge, and financial support, as well as existing governance structures. Existing ocean-governance structures are already facing multi-dimensional, scale-related challenges because of climate change.

正如我们所看到的，微调后的模型提供了更全面的响应！这与来自ragas的增加的可信度得分相一致，因为答案更能代表检索到的上下文。

结论¶

因此，总的来说，仅使用大约61个问题进行微调实际上有助于提高我们的评估分数！

答案相关性：0.9725 -> 0.9607

答案相关性略有下降，但下降幅度非常小。

忠实度：0.7325 -> 0.7917

忠实度似乎有所改善！这意味着给出的答案更好地满足了最初提出的问题。