Adafactor
Adafactor class
keras.optimizers.Adafactor(
learning_rate=0.001,
beta_2_decay=-0.8,
epsilon_1=1e-30,
epsilon_2=0.001,
clip_threshold=1.0,
relative_step=True,
weight_decay=None,
clipnorm=None,
clipvalue=None,
global_clipnorm=None,
use_ema=False,
ema_momentum=0.99,
ema_overwrite_frequency=None,
loss_scale_factor=None,
gradient_accumulation_steps=None,
name="adafactor",
**kwargs
)
实现Adafactor算法的优化器.
Adafactor通常用于NLP任务,并且具有占用较少内存的优势,因为它只保存先前梯度的部分信息.
默认参数设置基于原始论文（见参考文献）.当梯度的维度大于2时,Adafactor优化器将在其累加器变量中分别删除最后两个维度.
参数:
learning_rate: 一个浮点数,一个
keras.optimizers.schedules.LearningRateSchedule实例,或
一个不带参数并返回实际使用值的调用.学习率.默认为0.001.
beta_2_decay: 浮点数,默认为-0.8.beta_2的衰减率.
epsilon_1: 浮点数,默认为1e-30.一个小的偏移量,以保持分母不为0.
epsilon_2: 浮点数,默认为1e-3.一个小的偏移量,以避免学习率随时间变得过小.
clip_threshold: 浮点数,默认为1.0.剪切阈值.这是Adafactor算法的一部分,独立于clipnorm、
clipvalue和global_clipnorm.
relative_step: 布尔值,默认为True.如果learning_rate是一个
常数且relative_step=True,学习率将根据当前迭代进行调整.这是Adafactor中默认的学习率衰减.
name: String. The name to use
for momentum accumulator weights created by
the optimizer.
weight_decay: Float. If set, weight decay is applied.
clipnorm: Float. If set, the gradient of each weight is individually
clipped so that its norm is no higher than this value.
clipvalue: Float. If set, the gradient of each weight is clipped to be
no higher than this value.
global_clipnorm: Float. If set, the gradient of all weights is clipped
so that their global norm is no higher than this value.
use_ema: Boolean, defaults to False.
If True, exponential moving average
(EMA) is applied. EMA consists of computing an exponential moving
average of the weights of the model (as the weight values change
after each training batch), and periodically overwriting the
weights with their moving average.
ema_momentum: Float, defaults to 0.99. Only used if use_ema=True.
This is the momentum to use when computing
the EMA of the model's weights:
new_average = ema_momentum * old_average + (1 - ema_momentum) *
current_variable_value.
ema_overwrite_frequency: Int or None, defaults to None. Only used if
use_ema=True. Every ema_overwrite_frequency steps of iterations,
we overwrite the model variable by its moving average.
If None, the optimizer
does not overwrite model variables in the middle of training,
and you need to explicitly overwrite the variables
at the end of training by calling
optimizer.finalize_variable_values() (which updates the model
variables in-place). When using the built-in fit() training loop,
this happens automatically after the last epoch,
and you don't need to do anything.
loss_scale_factor: Float or None. If a float, the scale factor will
be multiplied the loss before computing gradients, and the inverse
of the scale factor will be multiplied by the gradients before
updating variables. Useful for preventing underflow during
mixed precision training. Alternately,
keras.optimizers.LossScaleOptimizer will
automatically set a loss scale factor.
gradient_accumulation_steps: Int or None. If an int, model & optimizer
variables will not be updated at every step; instead they will be
updated every gradient_accumulation_steps steps, using the average
value of the gradients since the last update. This is known as
"gradient accumulation". This can be useful
when your batch size is very small, in order to reduce gradient
noise at each update step. EMA frequency will look at "accumulated"
iterations value (optimizer steps // gradient_accumulation_steps).
Learning rate schedules will look at "real" iterations value
(optimizer steps).
参考文献: