Table of Contents
Shortcuts
intermediate/ax_multiobjective_nas_tutorial

注意

点击 这里 下载完整的示例代码

使用Ax进行多目标NAS

创建于：2022年8月19日 | 最后更新：2024年7月31日 | 最后验证：2024年11月5日

作者： David Eriksson, Max Balandat, 以及Meta的自适应实验团队。

在本教程中，我们展示了如何使用Ax在流行的MNIST数据集上为一个简单的神经网络模型运行多目标神经架构搜索（NAS）。虽然底层方法通常用于更复杂的模型和更大的数据集，但我们选择了一个可以在笔记本电脑上轻松端到端运行且不超过20分钟的教程。

在许多NAS应用中，多个关注目标之间自然存在权衡。例如，在设备上部署模型时，我们可能希望最大化模型性能（例如，准确性），同时最小化竞争指标，如功耗、推理延迟或模型大小，以满足部署约束。通常，通过接受稍微降低的模型性能，我们可能能够大幅减少计算需求或预测延迟。探索这种权衡的原则性方法是可扩展和可持续AI的关键推动因素，并且在Meta有许多成功的应用——例如，请参阅我们关于自然语言理解模型的案例研究。

在我们的示例中，我们将调整两个隐藏层的宽度、学习率、dropout概率、批量大小和训练轮数。目标是在性能（验证集上的准确率）和模型大小（模型参数的数量）之间进行权衡。

本教程使用了以下PyTorch库：

  • PyTorch Lightning（指定模型和训练循环）

  • TorchX（用于远程/异步运行训练任务）

  • BoTorch（为Ax算法提供支持的贝叶斯优化库）

定义TorchX应用程序

我们的目标是优化在mnist_train_nas.py中定义的PyTorch Lightning训练任务。 为了使用TorchX实现这一点，我们编写了一个辅助函数，该函数接收训练任务的架构和超参数的值，并使用适当的设置创建一个TorchX AppDef。

from pathlib import Path

import torchx

from torchx import specs
from torchx.components import utils


def trainer(
    log_path: str,
    hidden_size_1: int,
    hidden_size_2: int,
    learning_rate: float,
    epochs: int,
    dropout: float,
    batch_size: int,
    trial_idx: int = -1,
) -> specs.AppDef:

    # define the log path so we can pass it to the TorchX ``AppDef``
    if trial_idx >= 0:
        log_path = Path(log_path).joinpath(str(trial_idx)).absolute().as_posix()

    return utils.python(
        # command line arguments to the training script
        "--log_path",
        log_path,
        "--hidden_size_1",
        str(hidden_size_1),
        "--hidden_size_2",
        str(hidden_size_2),
        "--learning_rate",
        str(learning_rate),
        "--epochs",
        str(epochs),
        "--dropout",
        str(dropout),
        "--batch_size",
        str(batch_size),
        # other config options
        name="trainer",
        script="mnist_train_nas.py",
        image=torchx.version.TORCHX_IMAGE,
    )

设置运行器

Ax的Runner抽象允许编写与各种后端的接口。Ax已经为TorchX提供了Runner，因此我们只需要配置它。在本教程中，我们以完全异步的方式在本地运行作业。

为了在集群上启动它们，您可以指定一个不同的TorchX调度器并适当调整配置。例如，如果您有一个Kubernetes集群，您只需要将调度器从local_cwd更改为kubernetes）。

import tempfile
from ax.runners.torchx import TorchXRunner

# Make a temporary dir to log our results into
log_dir = tempfile.mkdtemp()

ax_runner = TorchXRunner(
    tracker_base="/tmp/",
    component=trainer,
    # NOTE: To launch this job on a cluster instead of locally you can
    # specify a different scheduler and adjust arguments appropriately.
    scheduler="local_cwd",
    component_const_params={"log_path": log_dir},
    cfg={},
)

设置SearchSpace

首先，我们定义我们的搜索空间。Ax 支持整数和浮点类型的范围参数，以及可以具有非数值类型（如字符串）的选择参数。 我们将调整隐藏层大小、学习率、dropout 和 epoch 数量作为范围参数，并将批量大小调整为有序选择参数，以强制其为 2 的幂。

from ax.core import (
    ChoiceParameter,
    ParameterType,
    RangeParameter,
    SearchSpace,
)

parameters = [
    # NOTE: In a real-world setting, hidden_size_1 and hidden_size_2
    # should probably be powers of 2, but in our simple example this
    # would mean that ``num_params`` can't take on that many values, which
    # in turn makes the Pareto frontier look pretty weird.
    RangeParameter(
        name="hidden_size_1",
        lower=16,
        upper=128,
        parameter_type=ParameterType.INT,
        log_scale=True,
    ),
    RangeParameter(
        name="hidden_size_2",
        lower=16,
        upper=128,
        parameter_type=ParameterType.INT,
        log_scale=True,
    ),
    RangeParameter(
        name="learning_rate",
        lower=1e-4,
        upper=1e-2,
        parameter_type=ParameterType.FLOAT,
        log_scale=True,
    ),
    RangeParameter(
        name="epochs",
        lower=1,
        upper=4,
        parameter_type=ParameterType.INT,
    ),
    RangeParameter(
        name="dropout",
        lower=0.0,
        upper=0.5,
        parameter_type=ParameterType.FLOAT,
    ),
    ChoiceParameter(  # NOTE: ``ChoiceParameters`` don't require log-scale
        name="batch_size",
        values=[32, 64, 128, 256],
        parameter_type=ParameterType.INT,
        is_ordered=True,
        sort_values=True,
    ),
]

search_space = SearchSpace(
    parameters=parameters,
    # NOTE: In practice, it may make sense to add a constraint
    # hidden_size_2 <= hidden_size_1
    parameter_constraints=[],
)

设置指标

Ax 有一个 Metric 的概念，它定义了结果的属性以及如何获取这些结果的观察值。这使得例如可以编码如何从某些分布式执行后端获取数据，并在将其作为输入传递给 Ax 之前进行后处理。

在本教程中，我们将使用 多目标优化 以最大化验证准确性和最小化 模型参数数量为目标。后者代表模型延迟的简单代理， 对于小型机器学习模型来说，准确估计延迟是困难的 （在实际应用中，我们会在设备上运行模型时对延迟进行基准测试）。

在我们的示例中，TorchX将以完全异步的方式在本地运行训练任务，并将结果写入基于试验索引的log_dir（参见上面的trainer()函数）。我们将定义一个知道该日志目录的指标类。通过继承TensorboardCurveMetric，我们可以免费获得读取和解析TensorBoard日志的逻辑。

from ax.metrics.tensorboard import TensorboardMetric
from tensorboard.backend.event_processing import plugin_event_multiplexer as event_multiplexer

class MyTensorboardMetric(TensorboardMetric):

    # NOTE: We need to tell the new TensorBoard metric how to get the id /
    # file handle for the TensorBoard logs from a trial. In this case
    # our convention is to just save a separate file per trial in
    # the prespecified log dir.
    def _get_event_multiplexer_for_trial(self, trial):
        mul = event_multiplexer.EventMultiplexer(max_reload_threads=20)
        mul.AddRunsFromDirectory(Path(log_dir).joinpath(str(trial.index)).as_posix(), None)
        mul.Reload()

        return mul

    # This indicates whether the metric is queryable while the trial is
    # still running. We don't use this in the current tutorial, but Ax
    # utilizes this to implement trial-level early-stopping functionality.
    @classmethod
    def is_available_while_running(cls):
        return False

现在我们可以实例化准确性和模型参数数量的指标。这里curve_name是TensorBoard日志中指标的名称，而name是Ax内部使用的指标名称。我们还指定了lower_is_better以指示这两个指标的有利方向。

val_acc = MyTensorboardMetric(
    name="val_acc",
    tag="val_acc",
    lower_is_better=False,
)
model_num_params = MyTensorboardMetric(
    name="num_params",
    tag="num_params",
    lower_is_better=True,
)

设置OptimizationConfig

告诉Ax它应该优化的方式是通过一个 OptimizationConfig。 这里我们使用一个MultiObjectiveOptimizationConfig，因为我们将 进行多目标优化。

此外，Ax支持通过指定目标阈值对不同指标施加约束，这些阈值界定了我们希望在结果空间中探索的兴趣区域。对于这个例子，我们将约束验证准确率至少为0.94（94%），并且模型参数的数量最多为80,000。

from ax.core import MultiObjective, Objective, ObjectiveThreshold
from ax.core.optimization_config import MultiObjectiveOptimizationConfig


opt_config = MultiObjectiveOptimizationConfig(
    objective=MultiObjective(
        objectives=[
            Objective(metric=val_acc, minimize=False),
            Objective(metric=model_num_params, minimize=True),
        ],
    ),
    objective_thresholds=[
        ObjectiveThreshold(metric=val_acc, bound=0.94, relative=False),
        ObjectiveThreshold(metric=model_num_params, bound=80_000, relative=False),
    ],
)

创建Ax实验

在Ax中，Experiment对象是存储所有关于问题设置信息的对象。

from ax.core import Experiment

experiment = Experiment(
    name="torchx_mnist",
    search_space=search_space,
    optimization_config=opt_config,
    runner=ax_runner,
)

选择生成策略

一个GenerationStrategy是我们希望如何执行优化的抽象表示。虽然这可以自定义（如果您想这样做，请参阅本教程），但在大多数情况下，Ax可以根据搜索空间、优化配置和我们希望运行的总试验次数自动确定适当的策略。

通常，Ax 在开始基于模型的贝叶斯优化策略之前，会选择评估一些随机配置。

total_trials = 48  # total evaluation budget

from ax.modelbridge.dispatch_utils import choose_generation_strategy

gs = choose_generation_strategy(
    search_space=experiment.search_space,
    optimization_config=experiment.optimization_config,
    num_trials=total_trials,
  )

[INFO 01-02 21:56:00] ax.modelbridge.dispatch_utils: Using Models.BOTORCH_MODULAR since there is at least one ordered parameter and there are no unordered categorical parameters.
[INFO 01-02 21:56:00] ax.modelbridge.dispatch_utils: Calculating the number of remaining initialization trials based on num_initialization_trials=None max_initialization_trials=None num_tunable_parameters=6 num_trials=48 use_batch_trials=False
[INFO 01-02 21:56:00] ax.modelbridge.dispatch_utils: calculated num_initialization_trials=9
[INFO 01-02 21:56:00] ax.modelbridge.dispatch_utils: num_completed_initialization_trials=0 num_remaining_initialization_trials=9
[INFO 01-02 21:56:00] ax.modelbridge.dispatch_utils: `verbose`, `disable_progbar`, and `jit_compile` are not yet supported when using `choose_generation_strategy` with ModularBoTorchModel, dropping these arguments.
[INFO 01-02 21:56:00] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+BoTorch', steps=[Sobol for 9 trials, BoTorch for subsequent trials]). Iterations after 9 will take longer to generate due to model-fitting.

配置调度器

Scheduler 作为优化的循环控制。它与后端通信以启动试验、检查其状态并检索结果。在本教程的情况下，它只是读取和解析本地保存的日志。在远程执行设置中，它将调用API。以下来自Ax Scheduler教程的插图总结了Scheduler如何与用于运行试验评估的外部系统交互：

../_static/img/ax_scheduler_illustration.png

Scheduler 需要 Experiment 和 GenerationStrategy。 可以通过 SchedulerOptions 传递一组选项。在这里， 我们配置了总评估次数以及 max_pending_trials， 即应同时运行的最大试验次数。在我们的本地设置中， 这是作为独立进程运行的训练作业的数量，而在远程执行设置中， 这将是您希望并行使用的机器数量。

from ax.service.scheduler import Scheduler, SchedulerOptions

scheduler = Scheduler(
    experiment=experiment,
    generation_strategy=gs,
    options=SchedulerOptions(
        total_trials=total_trials, max_pending_trials=4
    ),
)

[WARNING 01-02 21:56:00] ax.service.utils.with_db_settings_base: Ax currently requires a sqlalchemy version below 2.0. This will be addressed in a future release. Disabling SQL storage in Ax for now, if you would like to use SQL storage please install Ax with mysql extras via `pip install ax-platform[mysql]`.
[INFO 01-02 21:56:00] Scheduler: `Scheduler` requires experiment to have immutable search space and optimization config. Setting property immutable_search_space_and_opt_config to `True` on experiment.

运行优化

现在一切都已经配置好了，我们可以让Ax以完全自动化的方式运行优化。调度器将定期检查日志以获取所有当前运行的试验的状态，如果一个试验完成，调度器将更新其在实验中的状态，并获取贝叶斯优化算法所需的观测数据。

scheduler.run_all_trials()

[INFO 01-02 21:56:00] Scheduler: Fetching data for newly completed trials: [].
/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 01-02 21:56:00] Scheduler: Running trials [0]...
/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 01-02 21:56:01] Scheduler: Running trials [1]...
/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 01-02 21:56:02] Scheduler: Running trials [2]...
/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 01-02 21:56:03] Scheduler: Running trials [3]...
[INFO 01-02 21:56:04] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:04] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 01-02 21:56:05] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:05] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 01-02 21:56:07] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:07] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 01-02 21:56:09] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:09] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 01-02 21:56:12] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:12] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 4).
[INFO 01-02 21:56:17] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:17] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 4).
[INFO 01-02 21:56:25] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:25] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 4).
[INFO 01-02 21:56:36] Scheduler: Fetching data for newly completed trials: [1, 3].
[INFO 01-02 21:56:36] Scheduler: Retrieved COMPLETED trials: [1, 3].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 01-02 21:56:36] Scheduler: Running trials [4]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 01-02 21:56:37] Scheduler: Running trials [5]...
[INFO 01-02 21:56:38] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:38] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 01-02 21:56:39] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:39] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 01-02 21:56:41] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:41] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 01-02 21:56:43] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:43] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 01-02 21:56:47] Scheduler: Fetching data for newly completed trials: [2].
[INFO 01-02 21:56:47] Scheduler: Retrieved COMPLETED trials: [2].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 01-02 21:56:47] Scheduler: Running trials [6]...
[INFO 01-02 21:56:48] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:48] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 01-02 21:56:49] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:49] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 01-02 21:56:50] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:50] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 01-02 21:56:52] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:52] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 01-02 21:56:56] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:56:56] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 4).
[INFO 01-02 21:57:01] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:01] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 4).
[INFO 01-02 21:57:08] Scheduler: Fetching data for newly completed trials: [4].
[INFO 01-02 21:57:08] Scheduler: Retrieved COMPLETED trials: [4].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 01-02 21:57:09] Scheduler: Running trials [7]...
[INFO 01-02 21:57:10] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:10] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 01-02 21:57:11] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:11] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 01-02 21:57:12] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:12] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 01-02 21:57:14] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:14] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 01-02 21:57:18] Scheduler: Fetching data for newly completed trials: [6].
[INFO 01-02 21:57:18] Scheduler: Retrieved COMPLETED trials: [6].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 01-02 21:57:18] Scheduler: Running trials [8]...
[INFO 01-02 21:57:19] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:19] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 01-02 21:57:20] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:20] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 01-02 21:57:21] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:21] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 01-02 21:57:24] Scheduler: Fetching data for newly completed trials: [0, 5].
[INFO 01-02 21:57:24] Scheduler: Retrieved COMPLETED trials: [0, 5].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:57:27] Scheduler: Running trials [9]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:57:33] Scheduler: Running trials [10]...
[INFO 01-02 21:57:34] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:34] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 01-02 21:57:35] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:35] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 01-02 21:57:37] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:37] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 01-02 21:57:39] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:39] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 01-02 21:57:42] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:42] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 4).
[INFO 01-02 21:57:47] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:47] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 4).
[INFO 01-02 21:57:55] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:57:55] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 4).
[INFO 01-02 21:58:06] Scheduler: Fetching data for newly completed trials: [10].
[INFO 01-02 21:58:06] Scheduler: Retrieved COMPLETED trials: [10].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:58:14] Scheduler: Running trials [11]...
[INFO 01-02 21:58:15] Scheduler: Fetching data for newly completed trials: 8 - 9.
[INFO 01-02 21:58:15] Scheduler: Retrieved COMPLETED trials: 8 - 9.
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:58:23] Scheduler: Running trials [12]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:58:30] Scheduler: Running trials [13]...
[INFO 01-02 21:58:31] Scheduler: Fetching data for newly completed trials: [7].
[INFO 01-02 21:58:31] Scheduler: Retrieved COMPLETED trials: [7].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:58:33] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:58:33] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 21:58:34] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:58:34] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 21:58:35] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:58:35] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 21:58:38] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:58:38] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 21:58:41] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:58:41] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 21:58:46] Scheduler: Fetching data for newly completed trials: 11 - 12.
[INFO 01-02 21:58:46] Scheduler: Retrieved COMPLETED trials: 11 - 12.
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:58:51] Scheduler: Running trials [14]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:58:58] Scheduler: Running trials [15]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:59:04] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 21:59:04] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:04] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 21:59:05] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:05] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 21:59:06] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:06] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 21:59:09] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:09] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 21:59:12] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:12] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 21:59:17] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:17] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 01-02 21:59:25] Scheduler: Fetching data for newly completed trials: [13].
[INFO 01-02 21:59:25] Scheduler: Retrieved COMPLETED trials: [13].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:59:31] Scheduler: Running trials [16]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:59:36] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 21:59:36] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:36] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 21:59:37] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:37] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 21:59:39] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:39] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 21:59:41] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 21:59:41] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 21:59:44] Scheduler: Fetching data for newly completed trials: [15].
[INFO 01-02 21:59:44] Scheduler: Retrieved COMPLETED trials: [15].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:59:52] Scheduler: Running trials [17]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 21:59:57] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 21:59:57] Scheduler: Fetching data for newly completed trials: [14].
[INFO 01-02 21:59:57] Scheduler: Retrieved COMPLETED trials: [14].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:00:05] Scheduler: Running trials [18]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:00:11] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:00:11] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:00:11] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:00:12] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:00:12] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:00:13] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:00:13] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:00:16] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:00:16] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:00:19] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:00:19] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:00:24] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:00:24] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 01-02 22:00:32] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:00:32] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 3).
[INFO 01-02 22:00:43] Scheduler: Fetching data for newly completed trials: [16].
[INFO 01-02 22:00:43] Scheduler: Retrieved COMPLETED trials: [16].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:00:54] Scheduler: Running trials [19]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:01:00] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:01:00] Scheduler: Fetching data for newly completed trials: [17].
[INFO 01-02 22:01:00] Scheduler: Retrieved COMPLETED trials: [17].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:01:07] Scheduler: Running trials [20]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:01:12] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:01:12] Scheduler: Fetching data for newly completed trials: [18].
[INFO 01-02 22:01:12] Scheduler: Retrieved COMPLETED trials: [18].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:01:24] Scheduler: Running trials [21]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:01:30] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:01:30] Scheduler: Fetching data for newly completed trials: [19].
[INFO 01-02 22:01:30] Scheduler: Retrieved COMPLETED trials: [19].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:01:47] Scheduler: Running trials [22]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:01:53] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:01:53] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:01:53] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:01:54] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:01:54] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:01:56] Scheduler: Fetching data for newly completed trials: [21].
[INFO 01-02 22:01:56] Scheduler: Retrieved COMPLETED trials: [21].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:02:13] Scheduler: Running trials [23]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:02:16] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:02:16] Scheduler: Fetching data for newly completed trials: [20, 22].
[INFO 01-02 22:02:16] Scheduler: Retrieved COMPLETED trials: [20, 22].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:02:29] Scheduler: Running trials [24]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:02:44] Scheduler: Running trials [25]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:02:53] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:02:53] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:02:53] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:02:54] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:02:54] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:02:56] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:02:56] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:02:58] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:02:58] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:03:02] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:03:02] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:03:07] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:03:07] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 01-02 22:03:14] Scheduler: Fetching data for newly completed trials: [23].
[INFO 01-02 22:03:14] Scheduler: Retrieved COMPLETED trials: [23].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:03:24] Scheduler: Running trials [26]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:03:28] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:03:28] Scheduler: Fetching data for newly completed trials: [24].
[INFO 01-02 22:03:28] Scheduler: Retrieved COMPLETED trials: [24].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:03:44] Scheduler: Running trials [27]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:03:52] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:03:52] Scheduler: Fetching data for newly completed trials: [25].
[INFO 01-02 22:03:52] Scheduler: Retrieved COMPLETED trials: [25].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:04:14] Scheduler: Running trials [28]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:04:25] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:04:25] Scheduler: Fetching data for newly completed trials: [26].
[INFO 01-02 22:04:25] Scheduler: Retrieved COMPLETED trials: [26].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:04:44] Scheduler: Running trials [29]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:04:55] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:04:55] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:04:55] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:04:56] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:04:56] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:04:57] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:04:57] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:04:59] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:04:59] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:05:03] Scheduler: Fetching data for newly completed trials: [27].
[INFO 01-02 22:05:03] Scheduler: Retrieved COMPLETED trials: [27].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:05:18] Scheduler: Running trials [30]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:05:27] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:05:27] Scheduler: Fetching data for newly completed trials: [28].
[INFO 01-02 22:05:27] Scheduler: Retrieved COMPLETED trials: [28].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:05:37] Scheduler: Running trials [31]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:05:45] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:05:45] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:05:45] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:05:46] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:05:46] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:05:47] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:05:47] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:05:50] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:05:50] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:05:53] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:05:53] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:05:58] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:05:58] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 01-02 22:06:06] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:06:06] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 3).
[INFO 01-02 22:06:17] Scheduler: Fetching data for newly completed trials: 29 - 30.
[INFO 01-02 22:06:17] Scheduler: Retrieved COMPLETED trials: 29 - 30.
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:06:26] Scheduler: Running trials [32]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:06:35] Scheduler: Running trials [33]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:06:43] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:06:43] Scheduler: Fetching data for newly completed trials: [31].
[INFO 01-02 22:06:43] Scheduler: Retrieved COMPLETED trials: [31].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:06:55] Scheduler: Running trials [34]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:07:04] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:07:04] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:07:04] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:07:05] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:07:05] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:07:06] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:07:06] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:07:09] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:07:09] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:07:12] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:07:12] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:07:17] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:07:17] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 01-02 22:07:25] Scheduler: Fetching data for newly completed trials: [32].
[INFO 01-02 22:07:25] Scheduler: Retrieved COMPLETED trials: [32].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:07:38] Scheduler: Running trials [35]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:07:48] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:07:48] Scheduler: Fetching data for newly completed trials: [33].
[INFO 01-02 22:07:48] Scheduler: Retrieved COMPLETED trials: [33].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:07:59] Scheduler: Running trials [36]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:08:08] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:08:08] Scheduler: Fetching data for newly completed trials: [34].
[INFO 01-02 22:08:08] Scheduler: Retrieved COMPLETED trials: [34].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:08:17] Scheduler: Running trials [37]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:08:24] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:08:24] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:08:24] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:08:25] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:08:25] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:08:26] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:08:26] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:08:29] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:08:29] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:08:32] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:08:32] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:08:37] Scheduler: Fetching data for newly completed trials: [35].
[INFO 01-02 22:08:37] Scheduler: Retrieved COMPLETED trials: [35].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:08:47] Scheduler: Running trials [38]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:08:54] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:08:54] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:08:54] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:08:55] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:08:55] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:08:56] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:08:56] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:08:58] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:08:58] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:09:02] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:09:02] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:09:07] Scheduler: Fetching data for newly completed trials: [36].
[INFO 01-02 22:09:07] Scheduler: Retrieved COMPLETED trials: [36].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:09:18] Scheduler: Running trials [39]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:09:23] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:09:23] Scheduler: Fetching data for newly completed trials: [37].
[INFO 01-02 22:09:23] Scheduler: Retrieved COMPLETED trials: [37].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:09:36] Scheduler: Running trials [40]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:09:43] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:09:43] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:09:43] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:09:44] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:09:44] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:09:46] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:09:46] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:09:48] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:09:48] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:09:51] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:09:51] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:09:56] Scheduler: Fetching data for newly completed trials: [38].
[INFO 01-02 22:09:56] Scheduler: Retrieved COMPLETED trials: [38].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:10:08] Scheduler: Running trials [41]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:10:16] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:10:16] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:10:16] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:10:17] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:10:17] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:10:18] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:10:18] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:10:21] Scheduler: Fetching data for newly completed trials: [39].
[INFO 01-02 22:10:21] Scheduler: Retrieved COMPLETED trials: [39].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:10:40] Scheduler: Running trials [42]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:10:55] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:10:55] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:10:55] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:10:56] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:10:56] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:10:57] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:10:57] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:10:59] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:10:59] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:11:03] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:11:03] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:11:08] Scheduler: Fetching data for newly completed trials: 40 - 41.
[INFO 01-02 22:11:08] Scheduler: Retrieved COMPLETED trials: 40 - 41.
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:11:15] Scheduler: Running trials [43]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:11:24] Scheduler: Running trials [44]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:11:29] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:11:29] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:11:29] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:11:30] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:11:30] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:11:31] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:11:31] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:11:34] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:11:34] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:11:37] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:11:37] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:11:42] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:11:42] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 01-02 22:11:50] Scheduler: Fetching data for newly completed trials: [42].
[INFO 01-02 22:11:50] Scheduler: Retrieved COMPLETED trials: [42].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:12:01] Scheduler: Running trials [45]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:12:07] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:12:07] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:07] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:12:08] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:08] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:12:10] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:10] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:12:12] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:12] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:12:16] Scheduler: Fetching data for newly completed trials: [43].
[INFO 01-02 22:12:16] Scheduler: Retrieved COMPLETED trials: [43].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:12:27] Scheduler: Running trials [46]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:12:31] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 01-02 22:12:31] Scheduler: Fetching data for newly completed trials: [44].
[INFO 01-02 22:12:31] Scheduler: Retrieved COMPLETED trials: [44].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:12:40] Scheduler: Running trials [47]...
[INFO 01-02 22:12:40] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:40] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 01-02 22:12:41] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:41] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 01-02 22:12:43] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:43] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 01-02 22:12:45] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:45] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 01-02 22:12:49] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:49] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 01-02 22:12:54] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:12:54] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 01-02 22:13:01] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:01] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 3).
[INFO 01-02 22:13:13] Scheduler: Fetching data for newly completed trials: [45].
[INFO 01-02 22:13:13] Scheduler: Retrieved COMPLETED trials: [45].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:13:13] Scheduler: Done submitting trials, waiting for remaining 2 running trials...
[INFO 01-02 22:13:13] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:13] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 2).
[INFO 01-02 22:13:14] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:14] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 2).
[INFO 01-02 22:13:15] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:15] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 2).
[INFO 01-02 22:13:18] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:18] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 2).
[INFO 01-02 22:13:21] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:21] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 2).
[INFO 01-02 22:13:26] Scheduler: Fetching data for newly completed trials: [46].
[INFO 01-02 22:13:26] Scheduler: Retrieved COMPLETED trials: [46].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 01-02 22:13:26] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:26] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 1).
[INFO 01-02 22:13:27] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:27] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 1).
[INFO 01-02 22:13:29] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:29] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 1).
[INFO 01-02 22:13:31] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:31] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 1).
[INFO 01-02 22:13:34] Scheduler: Fetching data for newly completed trials: [].
[INFO 01-02 22:13:34] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 1).
[INFO 01-02 22:13:39] Scheduler: Fetching data for newly completed trials: [47].
[INFO 01-02 22:13:39] Scheduler: Retrieved COMPLETED trials: [47].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.


OptimizationResult()

评估结果

我们现在可以使用Ax中包含的辅助函数和可视化工具来检查优化的结果。

首先，我们生成一个包含实验结果摘要的数据框。此数据框中的每一行对应一个试验（即运行的训练任务），并包含有关试验状态、评估的参数配置以及观察到的指标值的信息。这提供了一种简单的方法来检查优化的合理性。

from ax.service.utils.report_utils import exp_to_df

df = exp_to_df(experiment)
df.head(10)

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[WARNING 01-02 22:13:39] ax.service.utils.report_utils: Column reason missing for all trials. Not appending column.
trial_index arm_name trial_status generation_method is_feasible num_params val_acc hidden_size_1 hidden_size_2 learning_rate epochs dropout batch_size
0 0 0_0 COMPLETED Sobol False 16810.0 0.892440 19 66 0.003182 4 0.190970 32
1 1 1_0 COMPLETED Sobol False 40280.0 0.916779 50 18 0.000614 2 0.425028 128
2 2 2_0 COMPLETED Sobol False 112720.0 0.963429 127 96 0.001233 3 0.334354 256
3 3 3_0 COMPLETED Sobol False 30653.0 0.905993 37 35 0.000234 1 0.037933 64
4 4 4_0 COMPLETED Sobol False 26056.0 0.917627 28 108 0.000339 1 0.062996 32
5 5 5_0 COMPLETED Sobol True 71312.0 0.951953 87 32 0.006412 3 0.297078 128
6 6 6_0 COMPLETED Sobol False 50051.0 0.862612 59 55 0.000116 2 0.458433 256
7 7 7_0 COMPLETED Sobol False 19530.0 0.936613 24 21 0.002236 4 0.161959 64
8 8 8_0 COMPLETED Sobol False 18080.0 0.905648 20 80 0.000163 3 0.388082 64
9 9 9_0 COMPLETED BoTorch True 46403.0 0.952467 53 77 0.002655 3 0.263433 256


我们还可以可视化验证准确率和模型参数数量之间的帕累托前沿权衡。

提示

Ax 使用 Plotly 生成交互式图表，这些图表允许您进行缩放、裁剪或悬停等操作，以查看图表组件的详细信息。试试看，如果您想了解更多，请查看可视化教程。

最终的优化结果如下图所示，其中颜色对应于每次试验的迭代次数。我们看到我们的方法能够成功地探索权衡，并找到了具有高验证精度的大型模型以及具有相对较低验证精度的小型模型。

from ax.service.utils.report_utils import _pareto_frontier_scatter_2d_plotly

_pareto_frontier_scatter_2d_plotly(experiment)

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[WARNING 01-02 22:13:40] ax.service.utils.report_utils: Column reason missing for all trials. Not appending column.
0.940.9450.950.9550.960.9650.9710k20k30k40k50k60k70k80k
1iteration48Observed metric values with Pareto frontierval_accnum_params


为了更好地理解我们的代理模型对黑箱目标学到了什么，我们可以看一下留一法交叉验证的结果。由于我们的模型是高斯过程，它们不仅提供点预测，还提供这些预测的不确定性估计。一个好的模型意味着预测均值（图中的点）接近45度线，并且置信区间以预期的频率覆盖45度线（这里我们使用95%的置信区间，所以我们期望它们在95%的时间内包含真实观测值）。

如下图所示，模型大小（num_params）指标比验证准确率（val_acc）指标更容易建模。

from ax.modelbridge.cross_validation import compute_diagnostics, cross_validate
from ax.plot.diagnostic import interact_cross_validation_plotly
from ax.utils.notebook.plotting import init_notebook_plotting, render

cv = cross_validate(model=gs.model)  # The surrogate model is stored on the ``GenerationStrategy``
compute_diagnostics(cv)

interact_cross_validation_plotly(cv)
0.90.950.860.880.90.920.940.960.98
Cross-validationActual OutcomePredicted OutcomeShow CIval_acc▼Yes▼


我们还可以制作等高线图，以更好地理解不同的目标如何依赖于两个输入参数。在下图中，我们展示了模型预测的验证准确率作为两个隐藏层大小的函数。随着隐藏层大小的增加，验证准确率明显提高。

from ax.plot.contour import interact_contour_plotly

interact_contour_plotly(model=gs.model, metric_name="val_acc")
204060801001202030405060708090100110120204060801001202030405060708090100110120
In-sample0.930.940.950.960.0020.0030.0040.0050.006val_acchidden_size_1hidden_size_1hidden_size_2MeanStandard Errorx-param:y-param:hidden_size_1▲hidden_size_2▲


同样地，我们在下图中展示了模型参数数量作为隐藏大小的函数，并看到它也随着隐藏大小的增加而增加（对hidden_size_1的依赖更大）。

interact_contour_plotly(model=gs.model, metric_name="num_params")
204060801001202030405060708090100110120204060801001202030405060708090100110120
In-sample25k50k75k100k10002000num_paramshidden_size_1hidden_size_1hidden_size_2MeanStandard Errorx-param:y-param:hidden_size_1▲hidden_size_2▲


致谢

我们感谢TorchX团队（特别是Kiuk Chung和Tristan Rice）在将TorchX与Ax集成方面的帮助。

脚本的总运行时间： (17分钟 54.688秒)

Gallery generated by Sphinx-Gallery

为了分析流量并优化您的体验，我们在本网站上使用cookies。通过点击或浏览，您同意我们使用cookies。作为本网站的当前维护者，Facebook的Cookies政策适用。了解更多信息，包括可用的控制选项：Cookies Policy。