trw.hparams

Submodules

Package Contents

Classes

HyperParam

DiscreteMapping

Map discrete value to another discrete value

DiscreteValue

Discrete value. This can be useful to select one choice among many

DiscreteIntegrer

Represent an integer hyper-parameter

DiscreteBoolean

Represent a boolean hyper-parameter

ContinuousUniform

Represent a continuous hyper-parameter

HyperParameters

Holds a repository of hyper-parameters

ContinuousPower

Represent a continuous power hyper-parameter

HyperParametersOptimizerRandomSearchLocal

Random hyper parameter run on a single machine

HyperParametersOptimizerHyperband

Implementation of Hyperband: a novel bandit based approach to hyper-parameter optimization

Functions

store_loss_params(output_location, loss, infos, hyper_parameters)

Export the loss as self contained .pkl file to be analyzed later

load_loss_params(output_location)

reload the loss as self contained .pkl file to be analyzed later

analyse_hyperparameters(hprams_path_pattern, output_path, hparams_to_visualize=None, params_forest_n_estimators=5000, params_forest_max_features_ratio=0.6, top_k_covariance=5, create_graphs=True, verbose=True, dpi=300)

Importance hyper-pramaeter estimation using random forest regressors
class trw.hparams.HyperParam
class trw.hparams.DiscreteMapping(list_name_value, current_value)

Bases: HyperParam

Map discrete value to another discrete value

e.g., this can be useful to test activation function as hyper-parameter

set_value(self, value)
get_value(self)
random_value(self)
Returns

a random value

__repr__(self)

Return repr(self).

class trw.hparams.DiscreteValue(values, current_value)

Bases: HyperParam

Discrete value. This can be useful to select one choice among many

set_value(self, value)
get_value(self)
random_value(self)
Returns

a random value

__repr__(self)

Return repr(self).

class trw.hparams.DiscreteIntegrer(current_value, min_range, max_range)

Bases: HyperParam

Represent an integer hyper-parameter

set_value(self, value)
get_value(self)
random_value(self)
Returns

a random value

__repr__(self)

Return repr(self).

class trw.hparams.DiscreteBoolean(current_value)

Bases: HyperParam

Represent a boolean hyper-parameter

set_value(self, value)
get_value(self)
random_value(self)
Returns

a random value

__repr__(self)

Return repr(self).

class trw.hparams.ContinuousUniform(current_value, min_range, max_range)

Bases: HyperParam

Represent a continuous hyper-parameter

set_value(self, value)
get_value(self)
random_value(self)
Returns

a random value

__repr__(self)

Return repr(self).

class trw.hparams.HyperParameters(hparams=None)

Holds a repository of hyper-parameters

create(self, hparam_name, hparam)

Create an hyper parameter if it is not already present

Parameters

  • hparam_name – the name of the hyper-parameter to create

  • hparam – the hyper-parameter description and value

Returns

the hyper parameter value

generate_random_hparams(self)

Set hyper-parameter to a random value

get_value(self, name)

Return the current value of an hyper-parameter

__str__(self)

Return str(self).

__len__(self)
class trw.hparams.ContinuousPower(current_value, exponent_min, exponent_max)

Bases: HyperParam

Represent a continuous power hyper-parameter

This type of distribution can be useful to test e.g., learning rate hyper-parameter. Given a random number x generated from uniform interval (min_range, max_range), return 10 ** x

set_value(self, value)
get_value(self)
random_value(self)
__repr__(self)

Return repr(self).

class trw.hparams.HyperParametersOptimizerRandomSearchLocal(evaluate_hparams_fn, repeat, log_string=log_random, result_prefix='hparams-random')

Random hyper parameter run on a single machine

We need to define the hyper parameter evaluation function:

def evaluate_hparams(hparams):
    # evaluate an hyper-parameter configuration and return a loss value and some additional information
    # e.g., result report
    return 0.0, {}
optimize(self, result_path)

Optimize the hyper-parameter search using random search

Parameters

result_path – where to save the information of each run. Can be None, in this case nothing is exported.

Returns

the results of all the runs

trw.hparams.store_loss_params(output_location, loss, infos, hyper_parameters)

Export the loss as self contained .pkl file to be analyzed later

trw.hparams.load_loss_params(output_location)

reload the loss as self contained .pkl file to be analyzed later

class trw.hparams.HyperParametersOptimizerHyperband(evaluate_hparams_fn, result_prefix='', max_iter=81, eta=3, repeat=100, log_string=log_hyperband, always_include_default_hparams_in_each_cycle=True)

Implementation of Hyperband: a novel bandit based approach to hyper-parameter optimization https://arxiv.org/abs/1603.06560

def evaluate_hparams(hparams, nb_epochs):

# evaluate an hyper-parameter configuration and return a loss value and some additional information # e.g., result report return 0.0, {}

_repeat_one(self, result_path, hyper_parameters, repeat_id, nb_runs)

Run full Hyperband search :param result_path: :return: the last round of configurations

optimize(self, result_path)

Optimize the hyper parameters using Hyperband

Parameters

result_path – where to save the information of each run. Can be None, in this case nothing is exported.

Returns

the results of all the runs

trw.hparams.analyse_hyperparameters(hprams_path_pattern, output_path, hparams_to_visualize=None, params_forest_n_estimators=5000, params_forest_max_features_ratio=0.6, top_k_covariance=5, create_graphs=True, verbose=True, dpi=300)

Importance hyper-pramaeter estimation using random forest regressors

From simulation, the ordering of hyper-parameters importance is correct, but the importance value itself may be over-estimated (for the best param) and underestimated (for the others).

The scatter plot for each hparam is useful to understand in what direction the hyper-parameter should be modified

The covariance plot can be used to understand the relation between most important hyper-parameter

WARNING: [1] With correlated features, strong features can end up with low scores and the method can be biased towards variables with many categories. See for more details: see http://blog.datadive.net/selecting-good-features-part-iii-random-forests/ and https://link.springer.com/article/10.1186%2F1471-2105-8-25

Parameters

  • params_forest_n_estimators – number of trees used to estimate the loss from the hyperparameters

  • params_forest_max_features_ratio – the maximum number of features to be used. Note we don’t want to select all the features to limit the correlation importance decrease effect [1]

  • hprams_path_pattern – a pattern (globing) to be used to select the hyper parameter files

  • hparams_to_visualize – a list of hparam names to visualize or None. If None, display from the most important (i.e., causing the most loss variation) to the least

  • create_graphs – if True, export matplotlib visualizations

  • top_k_covariance – export the parameter covariance for the most important k hyper-parameters

  • output_path – where to export the graph

  • dpi – the resolution of the exported graph

  • verbose – if True, display additional information

Returns