► Keras 3 API 文档 / 优化器 / Adadelta

Adadelta

[source]

Adadelta class

keras.optimizers.Adadelta(
    learning_rate=0.001,
    rho=0.95,
    epsilon=1e-07,
    weight_decay=None,
    clipnorm=None,
    clipvalue=None,
    global_clipnorm=None,
    use_ema=False,
    ema_momentum=0.99,
    ema_overwrite_frequency=None,
    loss_scale_factor=None,
    gradient_accumulation_steps=None,
    name="adadelta",
    **kwargs
)

优化器实现了Adadelta算法.

Adadelta优化是一种随机梯度下降方法,基于每个维度的自适应学习率来解决两个缺点:

  • 训练过程中学习率的持续衰减.
  • 需要手动选择的全局学习率.

Adadelta是Adagrad的一个更强大的扩展,它根据梯度更新的移动窗口调整学习率,而不是累积所有过去的梯度.这样,即使在进行了许多更新之后,Adadelta仍然可以继续学习.与Adagrad相比,在Adadelta的原始版本中,您不需要设置初始学习率.在这个版本中,可以像大多数其他Keras优化器一样设置初始学习率.

参数: learning_rate: 一个浮点数,一个 keras.optimizers.schedules.LearningRateSchedule实例,或 一个不带参数并返回实际值的回调函数.学习率.默认为0.001.注意,Adadelta 往往受益于比其他优化器更高的初始学习率值.要匹配原始论文的确切形式, 请使用1.0. rho: 一个浮点数值.衰减率.默认为0.95. epsilon: 用于维护数值稳定性的小浮点数值. name: String. The name to use for momentum accumulator weights created by the optimizer. weight_decay: Float. If set, weight decay is applied. clipnorm: Float. If set, the gradient of each weight is individually clipped so that its norm is no higher than this value. clipvalue: Float. If set, the gradient of each weight is clipped to be no higher than this value. global_clipnorm: Float. If set, the gradient of all weights is clipped so that their global norm is no higher than this value. use_ema: Boolean, defaults to False. If True, exponential moving average (EMA) is applied. EMA consists of computing an exponential moving average of the weights of the model (as the weight values change after each training batch), and periodically overwriting the weights with their moving average. ema_momentum: Float, defaults to 0.99. Only used if use_ema=True. This is the momentum to use when computing the EMA of the model's weights: new_average = ema_momentum * old_average + (1 - ema_momentum) * current_variable_value. ema_overwrite_frequency: Int or None, defaults to None. Only used if use_ema=True. Every ema_overwrite_frequency steps of iterations, we overwrite the model variable by its moving average. If None, the optimizer does not overwrite model variables in the middle of training, and you need to explicitly overwrite the variables at the end of training by calling optimizer.finalize_variable_values() (which updates the model variables in-place). When using the built-in fit() training loop, this happens automatically after the last epoch, and you don't need to do anything. loss_scale_factor: Float or None. If a float, the scale factor will be multiplied the loss before computing gradients, and the inverse of the scale factor will be multiplied by the gradients before updating variables. Useful for preventing underflow during mixed precision training. Alternately, keras.optimizers.LossScaleOptimizer will automatically set a loss scale factor. gradient_accumulation_steps: Int or None. If an int, model & optimizer variables will not be updated at every step; instead they will be updated every gradient_accumulation_steps steps, using the average value of the gradients since the last update. This is known as "gradient accumulation". This can be useful when your batch size is very small, in order to reduce gradient noise at each update step. EMA frequency will look at "accumulated" iterations value (optimizer steps // gradient_accumulation_steps). Learning rate schedules will look at "real" iterations value (optimizer steps).

参考文献:

Adadelta
Adadelta class