trw.hparams

Submodules

• trw.hparams.creators
• trw.hparams.interpret_params
• trw.hparams.params
• trw.hparams.params_optimizer
• trw.hparams.params_optimizer_hyperband
• trw.hparams.params_optimizer_random_search
• trw.hparams.store

Package Contents

Classes

HyperParam	Represent an hyper-parameter
DiscreteMapping	Map discrete value to another discrete value
DiscreteValue	Discrete value. This can be useful to select one choice among many
DiscreteInteger	Represent an integer hyper-parameter
DiscreteBoolean	Represent a boolean hyper-parameter
ContinuousUniform	Represent a continuous hyper-parameter
HyperParameters	Holds a repository a set of hyper-parameters
ContinuousPower	Represent a continuous power hyper-parameter
HyperParameterRepository	Holds the current hyper-parameters
RunStore	Helper class that provides a standard way to create an ABC using
RunResult	Represent the result of a run
RunStoreFile	Linear file based store.
HyperParametersOptimizerRandomSearchLocal	hyper-parameter search using a random walk on a single machine
HyperParametersOptimizerHyperband	Implementation of Hyperband: a novel bandit based approach to hyper-parameter optimization 3

Functions

create_discrete_value(name: str, default_value: Any, values: List[Any]) → Any
create_boolean(name: str, default_value: Any) → bool
create_discrete_integer(name: str, default_value: Any, min_range: int, max_range: int) → Any
create_continuous_power(name: str, default_value: float, exponent_min: float, exponent_max: float) → float
create_continuous_uniform(name: str, default_value: float, min_range: float, max_range: float) → float
create_discrete_mapping(name: str, default_value: Any, mapping: Dict[Any, Any]) → Any
analyse_hyperparameters(run_results: List[trw.hparams.store.RunResult], output_path: str, loss_fn: Callable[[trw.hparams.store.Metrics], float] = lambda metrics: metrics['loss'], hparams_to_visualize: List[str] = None, params_forest_n_estimators: int = 5000, params_forest_max_features_ratio: float = 0.6, top_k_covariance: int = 5, create_graphs: bool = True, verbose: bool = True, dpi: int = 300) → Dict[str, List]	Importance hyper-parameter estimation using random forest regressors.
create_optimizers_fn(datasets: trw.basic_typing.Datasets, model: torch.nn.Module, optimizers: Sequence[typing_extensions.Literal[adam, sgd]] = ('adam', 'sgd'), lr_range: Tuple[float, float, float] = (0.001, -5, -1), momentum: Sequence[float] = (0.5, 0.9, 0.99), beta_1: Sequence[float] = (0.9, ), beta_2: Sequence[float] = (0.999, 0.99), eps: Sequence[float] = (1e-08, ), weight_decay: Optional[Sequence[float]] = (0.0, 0.0001, 1e-05, 1e-06, 1e-08), name_prefix='trw.') → torch.optim.Optimizer	Create hyper-parameters for a wide range of optimizer search.
create_activation(name: str, default_value: torch.nn.Module, functions: Sequence[trw.basic_typing.ModuleCreator] = (nn.ReLU, nn.ReLU6, nn.LeakyReLU, nn.ELU, nn.PReLU, nn.RReLU, nn.SELU, nn.CELU, nn.Softplus)) → torch.nn.Module	Create activation functions
create_pool_type(name: str, default_value: trw.layers.layer_config.PoolType, pools: Sequence[trw.layers.layer_config.PoolType] = (PoolType.MaxPool, PoolType.AvgPool, PoolType.FractionalMaxPool)) → trw.layers.layer_config.PoolType	Create a pooling type hyper-parameter
create_norm_type(name: str, default_value: Optional[trw.layers.layer_config.NormType], norms: Sequence[Optional[trw.layers.layer_config.NormType]] = (NormType.BatchNorm, NormType.InstanceNorm, None)) → trw.layers.layer_config.NormType	Create a normalization layer type hyper-parameter

class trw.hparams.HyperParam(name: str, default_value: Any, current_value: Any = None)

Bases: abc.ABC

Represent an hyper-parameter

set_value(self, value: Any) → None

Set the current value of an hyper-parameter

Parameters

value – the new current value

get_value(self) → Any

return the current value of an hyper-parameter

abstract randomize(self) → None

Randomize the current value of an hyper-parameter

class trw.hparams.DiscreteMapping(name: str, default_value: Any, mapping: Dict[Any, Any])

Bases: HyperParam

Map discrete value to another discrete value

e.g., this can be useful to test activation function as hyper-parameter

set_value(self, value: Any)

Set the current value of an hyper-parameter

Parameters

value – the new current value

get_value(self)

return the current value of an hyper-parameter

randomize(self)

Randomize the current value of an hyper-parameter

__repr__(self)

Return repr(self).

class trw.hparams.DiscreteValue(name: str, default_value: Any, values: List[Any])

Bases: HyperParam

Discrete value. This can be useful to select one choice among many

set_value(self, value: Any) → None

Set the current value of an hyper-parameter

Parameters

value – the new current value

randomize(self)

Randomize the current value of an hyper-parameter

__repr__(self)

Return repr(self).

class trw.hparams.DiscreteInteger(name: str, default_value: int, min_range: int, max_range: int)

Bases: HyperParam

Represent an integer hyper-parameter

randomize(self) → None

Randomize the current value of an hyper-parameter

__repr__(self)

Return repr(self).

class trw.hparams.DiscreteBoolean(name, default_value)

Bases: HyperParam

Represent a boolean hyper-parameter

randomize(self) → None

Randomize the current value of an hyper-parameter

__repr__(self)

Return repr(self).

class trw.hparams.ContinuousUniform(name: str, default_value: float, min_range: float, max_range: float)

Bases: HyperParam

Represent a continuous hyper-parameter

randomize(self) → None

Randomize the current value of an hyper-parameter

__repr__(self)

Return repr(self).

class trw.hparams.HyperParameters(hparams: Optional[Dict[str, HyperParam]] = None, randomize_at_creation: bool = False, hparams_to_randomize: Optional[List[str]] = None)

Holds a repository a set of hyper-parameters

hparam_to_be_randomized(self, haparam_name: str) → bool

create(self, hparam: HyperParam) → Any

Create an hyper parameter if it is not already present. If it is present, the given hparam is ignored

Parameters

hparam – the hyper-parameter

Returns

the hyper parameter value

randomize(self) → None

Set hyper-parameter to a random value

__getitem__(self, name: str)

get_value(self, name: str) → Any

Return the current value of an hyper-parameter

__str__(self)

Return str(self).

__repr__(self)

Return repr(self).

__len__(self)

class trw.hparams.ContinuousPower(name: str, default_value: float, exponent_min: float, exponent_max: float)

Bases: HyperParam

Represent a continuous power hyper-parameter

This type of distribution can be useful to test e.g., learning rate hyper-parameter. Given a random number x generated from uniform interval (min_range, max_range), return 10 ** x

Examples

>>> hp1 = ContinuousPower('hp1', default_value=0.1, exponent_min=-5, exponent_max=-1)
``hp1.get_value()`` would return a value in the range(1e-1, 1e-5)

randomize(self) → None

Randomize the current value of an hyper-parameter

__repr__(self)

Return repr(self).

class trw.hparams.HyperParameterRepository

Holds the current hyper-parameters

current_hparams :HyperParameters

static reset(new_hparams: Optional[HyperParameters] = None) → None

Replace the existing hyper parameters by a new one

Parameters

new_hparams – the new hyper-parameters

trw.hparams.create_discrete_value(name: str, default_value: Any, values: List[Any]) → Any

trw.hparams.create_boolean(name: str, default_value: Any) → bool

trw.hparams.create_discrete_integer(name: str, default_value: Any, min_range: int, max_range: int) → Any

trw.hparams.create_continuous_power(name: str, default_value: float, exponent_min: float, exponent_max: float) → float

trw.hparams.create_continuous_uniform(name: str, default_value: float, min_range: float, max_range: float) → float

trw.hparams.create_discrete_mapping(name: str, default_value: Any, mapping: Dict[Any, Any]) → Any

class trw.hparams.RunStore

Bases: abc.ABC

Helper class that provides a standard way to create an ABC using inheritance.

abstract close(self) → None

Close the store

abstract save_run(self, run_result: RunResult) → None

Save the results of a run

Parameters

run_result – the results to record

abstract load_all_runs(self) → Sequence[RunResult]

Load all the runs

__enter__(self)

__exit__(self, exc_type, exc_val, exc_tb)

class trw.hparams.RunResult(metrics: Metrics, hyper_parameters: trw.hparams.params.HyperParameters, history: trw.basic_typing.History, info: Any = None)

Represent the result of a run

class trw.hparams.RunStoreFile(store_location: str, serializer=pickle)

Bases: RunStore

Linear file based store.

Notes

• we don’t keep the file open, since we may have several stores reading from this location (single writer but many reader). Reading and writing must happen on the same thread

close(self) → None

Close the store

save_run(self, run_result: RunResult) → None

Save the results of a run

Parameters

run_result – the results to record

load_all_runs(self) → List[RunResult]

Load all the runs

class trw.hparams.HyperParametersOptimizerRandomSearchLocal(evaluate_fn: Callable[[trw.hparams.params.HyperParameters], Tuple[trw.hparams.store.Metrics, trw.basic_typing.History, Any]], repeat: int, log_string: Callable[[str], None] = log_with_logger)

Bases: trw.hparams.params_optimizer.HyperParametersOptimizer

hyper-parameter search using a random walk on a single machine

optimize(self, store: Optional[trw.hparams.store.RunStore] = None, hyper_parameters: Optional[trw.hparams.params.HyperParameters] = None)

Optimize the hyper-parameter search using random search

Parameters

• store – defines how the runs will be saved

• hyper_parameters – the hyper parameters to be optimized. If None, use the global repository trw.hparams.HyperParameterRepository

class trw.hparams.HyperParametersOptimizerHyperband(evaluate_fn: Callable[[trw.hparams.params.HyperParameters, float], Tuple[trw.hparams.store.Metrics, trw.basic_typing.History, Any]], loss_fn: Callable[[trw.hparams.store.Metrics], float], max_iter: int = 81, eta: int = 3, repeat: int = 100, log_string: Callable[[str], None] = log_hyperband, always_include_default_hparams_in_each_cycle: bool = True)

Bases: trw.hparams.params_optimizer.HyperParametersOptimizer

Implementation of Hyperband: a novel bandit based approach to hyper-parameter optimization [#]_

3

https://arxiv.org/abs/1603.06560

_repeat_one(self, repeat_id, nb_runs, hyper_parameters: trw.hparams.params.HyperParameters, store: Optional[trw.hparams.store.RunStore] = None) → Tuple[List[trw.hparams.store.RunResult], int]

Run full Hyperband search

Parameters

• repeat_id – the iteration number

• nb_runs – the run number

• store – how to store the result

• hyper_parameters – the hyper-parameters

Returns

a tuple of list of runs and number of runs for this iteration og hyperband

optimize(self, store: Optional[trw.hparams.store.RunStore], hyper_parameters: Optional[trw.hparams.params.HyperParameters] = None) → List[trw.hparams.store.RunResult]

Optimize the hyper parameters using Hyperband

Parameters

• store – how to result of each run. Can be None, in this case nothing is exported.

• hyper_parameters – the hyper parameters to be optimized. If None, use the global repository trw.hparams.HyperParameterRepository

Returns

the results of all the runs

trw.hparams.analyse_hyperparameters(run_results: List[trw.hparams.store.RunResult], output_path: str, loss_fn: Callable[[trw.hparams.store.Metrics], float] = lambda metrics: ..., hparams_to_visualize: List[str] = None, params_forest_n_estimators: int = 5000, params_forest_max_features_ratio: float = 0.6, top_k_covariance: int = 5, create_graphs: bool = True, verbose: bool = True, dpi: int = 300) → Dict[str, List]

Importance hyper-parameter estimation using random forest regressors.

From simulation, the ordering of hyper-parameters importance is correct, but the importance value itself may be over-estimated (for the best param) and underestimated (for the others).

The scatter plot for each hyper parameter is useful to understand in what direction the hyper-parameter should be modified.

The covariance plot can be used to understand the relation between most important hyper-parameter.

Warning

With correlated features, strong features can end up with low scores and the method can be biased towards variables with many categories. See for more details 1, 2.

1(1,2)

http://blog.datadive.net/selecting-good-features-part-iii-random-forests/

2

https://link.springer.com/article/10.1186%2F1471-2105-8-25

Parameters

• run_results – a list of runs

• output_path – where to export the graphs

• loss_fn – a function to extract a single value (loss) from a list of metrics

• hparams_to_visualize – a list of parameters (string) to visualize

• params_forest_n_estimators – number of trees used to estimate the loss from the hyperparameters

• params_forest_max_features_ratio – the maximum number of features to be used. Note we don’t want to select all the features to limit the correlation importance decrease effect 1

• top_k_covariance – export the parameter covariance for the most important k hyper-parameters

• create_graphs – if True, export matplotlib visualizations

• verbose – if True, display additional information

• dpi – the resolution of the exported graph

Returns

2 lists representing the hyper parameter name and importance

trw.hparams.create_optimizers_fn(datasets: trw.basic_typing.Datasets, model: torch.nn.Module, optimizers: Sequence[typing_extensions.Literal[adam, sgd]] = ('adam', 'sgd'), lr_range: Tuple[float, float, float] = (0.001, - 5, - 1), momentum: Sequence[float] = (0.5, 0.9, 0.99), beta_1: Sequence[float] = (0.9,), beta_2: Sequence[float] = (0.999, 0.99), eps: Sequence[float] = (1e-08,), weight_decay: Optional[Sequence[float]] = (0.0, 0.0001, 1e-05, 1e-06, 1e-08), name_prefix='trw.') → torch.optim.Optimizer

Create hyper-parameters for a wide range of optimizer search.

Hyper-parameters will be named using 2 groups of hyper-parameters: - trw.optimizers.*: most important hyper-parameters to search - trw.optimizers_fine.*: hyper-parameters that we might want to search but in most cases

would not significantly influence the results. These hyper-parameters maybe discarded during the hyper-parameter optimization

Parameters

• datasets – the datasets

• model – the model to be optimized

• optimizers – the optimizers to search

• lr_range – the learning rate range (min, max)

• momentum – the momentum values to test

• beta_1 – the beta_1 values to test

• beta_2 – the beta_2 values to test

• eps – the epsilon values to test

• weight_decay – the weight decay values to test

• name_prefix – prefix appended to the hyper-parameter name

Returns

A dict of optimizer per dataset

trw.hparams.create_activation(name: str, default_value: torch.nn.Module, functions: Sequence[trw.basic_typing.ModuleCreator] = (nn.ReLU, nn.ReLU6, nn.LeakyReLU, nn.ELU, nn.PReLU, nn.RReLU, nn.SELU, nn.CELU, nn.Softplus)) → torch.nn.Module

Create activation functions

Parameters

• name – the name of the hyper-parameter

• functions – the activation functions

• default_value – the default value at creation

Returns

a functor to create the activation function

trw.hparams.create_pool_type(name: str, default_value: trw.layers.layer_config.PoolType, pools: Sequence[trw.layers.layer_config.PoolType] = (PoolType.MaxPool, PoolType.AvgPool, PoolType.FractionalMaxPool)) → trw.layers.layer_config.PoolType

Create a pooling type hyper-parameter :param name: the name of the hyper-parameter :param pools: the available pooling types :param default_value: the default value at creation

Returns

a pooling type

trw.hparams.create_norm_type(name: str, default_value: Optional[trw.layers.layer_config.NormType], norms: Sequence[Optional[trw.layers.layer_config.NormType]] = (NormType.BatchNorm, NormType.InstanceNorm, None)) → trw.layers.layer_config.NormType

Create a normalization layer type hyper-parameter

Parameters

• name – the name of the hyper-parameter

• norms – a sequence of NormType

• default_value – the default value at creation

Returns

a normalization layer type