trw.callbacks.callback_learning_rate_finder

Module Contents

Classes

CallbackStopEpoch

Utility callback counting the number of samples. When maximum is reached, stop the iteration

CallbackLearningRateFinder

Identify a good range for the learning rate parameter.

Functions

plot_trend(export_path, lines_x, lines_y, title, xlabel, ylabel, y_scale='linear', x_scale='linear', maximum_chars_per_line=50, rotate_x=None, y_range=None, name_xy_markers=None)

Plot a graph defined by a list of x and y coordinates

default_identify_learning_rate_section(lines_x, lines_y, loss_ratio_to_discard=0.8)

Find a good section for the learning rate.

Attributes

logger

trw.callbacks.callback_learning_rate_finder.logger
class trw.callbacks.callback_learning_rate_finder.CallbackStopEpoch(nb_samples)

Utility callback counting the number of samples. When maximum is reached, stop the iteration

reset(self)
__call__(self, dataset_name, split_name, batch)
trw.callbacks.callback_learning_rate_finder.plot_trend(export_path, lines_x, lines_y, title, xlabel, ylabel, y_scale='linear', x_scale='linear', maximum_chars_per_line=50, rotate_x=None, y_range=None, name_xy_markers=None)

Plot a graph defined by a list of x and y coordinates

Parameters

  • export_path – folder where to export the figure

  • lines_x – a list of x coordinates

  • lines_y – a list of y coordinates

  • title – the title of the figure

  • xlabel – the label of axis x

  • ylabel – the label of axis y

  • y_scale – the scale of axis y

  • x_scale – the scale of axis x

  • maximum_chars_per_line – the maximum number of characters of the title per line

  • rotate_x – if True, the rotation angle of the label of the axis x

  • y_range – if not None, the (min, max) of the y-axis

  • name_xy_markers – a dictionary (name, (x, y)) of markers to be displayed

Returns

None

trw.callbacks.callback_learning_rate_finder.default_identify_learning_rate_section(lines_x, lines_y, loss_ratio_to_discard=0.8)

Find a good section for the learning rate.

Heuristic rules to find the best learning rate:

  1. worst loss is loss at epoch 0

  2. initially, the loss may not decrease due to small random variation, especially with small number of samples so tolerate that the initial LR may not be good

  3. after some epochs, the loss decrease to reach some minimum, then will increase significantly. Discard anything after this point

  4. find the LR achieving the minimum loss. This is our optimal LR

class trw.callbacks.callback_learning_rate_finder.CallbackLearningRateFinder(nb_samples_per_learning_rate=1000, learning_rate_start=1e-06, learning_rate_stop=10.0, learning_rate_mul=1.2, learning_rate_final_multiplier=0.8, dataset_name=None, split_name=None, dirname='lr_finder', identify_learning_rate_section=default_identify_learning_rate_section, set_new_learning_rate=False, param_maximum_loss_ratio=0.8)

Bases: trw.callbacks.callback.Callback

Identify a good range for the learning rate parameter.

See “Cyclical Learning Rates for Training Neural Networks”, Leslie N. Smith. https://arxiv.org/abs/1506.01186

Start from a small learning rate and every iteration, increase the learning rate by a factor. At the same time record the loss per epoch. Suitable learning rates will make the loss function decrease. We should select the highest learning rate which decreases the loss function.

__call__(self, options, history, model, losses, outputs, datasets, datasets_infos, callbacks_per_batch, **kwargs)

Note

The model will be deep copied so that we don’t influence the training

Parameters

**kwargs – required optimizers_fn