trw.utils

This package will contain utility function that only depends on numpy and pytorch

Submodules

Package Contents

Functions

upsample

clamp_n(tensor: torch.Tensor, min_values: Sequence[Any], max_values: Sequence[Any]) → torch.Tensor

Clamp a tensor with axis dependent values.

sub_tensor(tensor: torch.Tensor, min_indices: trw.basic_typing.Shape, max_indices_exclusive: trw.basic_typing.Shape) → torch.Tensor

Select a region of a tensor (without copy)

flatten(x: trw.basic_typing.TorchTensorNCX) → trw.basic_typing.TorchTensorNX

Flatten a tensor

global_max_pooling_2d(tensor: trw.basic_typing.TorchTensorNCX) → trw.basic_typing.TorchTensorNCX

2D Global max pooling.

global_average_pooling_2d(tensor: trw.basic_typing.TorchTensorNCX) → trw.basic_typing.TorchTensorNCX

2D Global average pooling.

global_average_pooling_3d(tensor: trw.basic_typing.TorchTensorNCX) → trw.basic_typing.TorchTensorNCX

3D Global average pooling.

global_max_pooling_3d(tensor: trw.basic_typing.TorchTensorNCX) → trw.basic_typing.TorchTensorNCX

3D Global max pooling.

batch_pad

batch_pad_joint(arrays: List[trw.basic_typing.TensorNCX], padding: trw.basic_typing.ShapeCX, mode: str = 'edge', constant_value: trw.basic_typing.Numeric = 0)

Add padding on a list of numpy or tensor array of samples. Supports arbitrary number of dimensions

batch_pad_torch(array: trw.basic_typing.TorchTensorNCX, padding: trw.basic_typing.ShapeCX, mode: str = 'edge', constant_value: trw.basic_typing.Numeric = 0)

Add padding on a numpy array of samples. This works for an arbitrary number of dimensions

batch_pad_numpy(array: trw.basic_typing.NumpyTensorNCX, padding: trw.basic_typing.ShapeCX, mode: str = 'edge', constant_value: trw.basic_typing.Numeric = 0)

Add padding on a numpy array of samples. This works for an arbitrary number of dimensions

batch_pad_minmax(array: trw.basic_typing.TensorNCX, padding_min: trw.basic_typing.ShapeCX, padding_max: trw.basic_typing.ShapeCX, mode: str = 'edge', constant_value: trw.basic_typing.Numeric = 0) → trw.basic_typing.TensorNCX

Add padding on a numpy array of samples. This works for an arbitrary number of dimensions

batch_pad_minmax_joint(arrays: List[trw.basic_typing.TensorNCX], padding_min: trw.basic_typing.ShapeCX, padding_max: trw.basic_typing.ShapeCX, mode: str = 'edge', constant_value: trw.basic_typing.Numeric = 0) → List[trw.basic_typing.TensorNCX]

Add padding on a list of numpy or tensor array of samples. Supports arbitrary number of dimensions

batch_pad_minmax_numpy(array: trw.basic_typing.NumpyTensorNCX, padding_min: trw.basic_typing.ShapeCX, padding_max: trw.basic_typing.ShapeCX, mode: str = 'edge', constant_value: trw.basic_typing.Numeric = 0) → trw.basic_typing.NumpyTensorNCX

Add padding on a numpy array of samples. This works for an arbitrary number of dimensions

batch_pad_minmax_torch(array: trw.basic_typing.TorchTensorNCX, padding_min: trw.basic_typing.ShapeCX, padding_max: trw.basic_typing.ShapeCX, mode: str = 'edge', constant_value: trw.basic_typing.Numeric = 0) → trw.basic_typing.TorchTensorNCX

Add padding on a numpy array of samples. This works for an arbitrary number of dimensions

safe_filename

optional_import

torch_requires(min_version: str, silent_fail: bool = False)

Requires a minimum pytorch version.

find_global_name(name)

Find a function or class from its name. If not found, raise a RuntimeError

bytes2human(n: Union[int, float]) → str

Format large number of bytes into readable string for a human

number2human(n: Union[int, float]) → str

Format large number into readable string for a human

collect_hierarchical_module_name(base_name, model, module_to_name=None)

Create a meaningful name of the module based on the module hierarchy

collect_hierarchical_parameter_name(base_name, model, parameter_to_name=None, with_grad_only=False)

Create a meaningful name of the module's parameters based on the module hierarchy

get_batch_n(split, nb_samples, indices, transforms, use_advanced_indexing)

Collect the split indices given and apply a series of transformations

to_value(v)

Convert where appropriate from tensors to numpy arrays

safe_lookup(dictionary, *keys, default=None)

Recursively access nested dictionaries

recursive_dict_update(dict, dict_update)

This adds any missing element from dict_update to dict, while keeping any key not

len_batch(batch)

param batch

a data split or a collections.Sequence

flatten_nested_dictionaries(d, root_name='', delimiter='-')

Recursively flatten a dictionary of arbitrary nested size into a flattened dictionary
trw.utils.upsample(tensor: trw.basic_typing.TensorNCX, size: trw.basic_typing.ShapeX, mode: typing_extensions.Literal[linear, nearest] = 'linear') → trw.basic_typing.TensorNCX

Upsample a 1D, 2D, 3D tensor

This is a wrapper around torch.nn.Upsample to make it more practical. Support integer based tensors.

Note

PyTorch as of version 1.3 doesn’t support non-floating point upsampling (see https://github.com/pytorch/pytorch/issues/13218 and https://github.com/pytorch/pytorch/issues/5580). Instead use a workaround (TODO assess the speed impact!).

Parameters

  • tensor – 1D (shape = b x c x n), 2D (shape = b x c x h x w) or 3D (shape = b x c x d x h x w)

  • size – if 1D, shape = n, if 2D shape = h x w, if 3D shape = d x h x w

  • mode – linear or nearest

Returns

an up-sampled tensor with same batch size and filter size as the input

trw.utils.clamp_n(tensor: torch.Tensor, min_values: Sequence[Any], max_values: Sequence[Any]) → torch.Tensor

Clamp a tensor with axis dependent values.

Parameters

  • tensor – a N-d torch.Tensor

  • min_values – a 1D torch.Tensor. Min value is axis dependent

  • max_values – a 1D torch.Tensor. Max value is axis dependent

Returns

tensor with values clamped to min_values and max_values

Examples

>>> t = torch.LongTensor([[1, 2, 3], [4, 5, 6]])
>>> min_values = torch.LongTensor([3, 2, 4])
>>> max_values = torch.LongTensor([3, 4, 8])
>>> clamped_t = clamp_n(t, min_values, max_values)
trw.utils.sub_tensor(tensor: torch.Tensor, min_indices: trw.basic_typing.Shape, max_indices_exclusive: trw.basic_typing.Shape) → torch.Tensor

Select a region of a tensor (without copy)

Examples

>>> t = torch.randn([5, 10])
>>> sub_t = sub_tensor(t, [2, 3], [4, 8])
Returns the t[2:4, 3:8]

>>> t = torch.randn([5, 10])
>>> sub_t = sub_tensor(t, [2], [4])
Returns the t[2:4]
Parameters

  • tensor – a tensor

  • min_indices – the minimum indices to select for each dimension

  • max_indices_exclusive – the maximum indices (excluded) to select for each dimension

Returns

torch.tensor

trw.utils.flatten(x: trw.basic_typing.TorchTensorNCX) → trw.basic_typing.TorchTensorNX

Flatten a tensor

Example, a tensor of shape[N, Z, Y, X] will be reshaped [N, Z * Y * X]

Parameters

x – a tensor

Returns: return a flattened tensor

trw.utils.global_max_pooling_2d(tensor: trw.basic_typing.TorchTensorNCX) → trw.basic_typing.TorchTensorNCX

2D Global max pooling.

Calculate the max value per sample per channel of a tensor.

Parameters

tensor – tensor with shape NCHW

Returns

a tensor of shape NC

trw.utils.global_average_pooling_2d(tensor: trw.basic_typing.TorchTensorNCX) → trw.basic_typing.TorchTensorNCX

2D Global average pooling.

Calculate the average value per sample per channel of a tensor.

Parameters

tensor – tensor with shape NCHW

Returns

a tensor of shape NC

trw.utils.global_average_pooling_3d(tensor: trw.basic_typing.TorchTensorNCX) → trw.basic_typing.TorchTensorNCX

3D Global average pooling.

Calculate the average value per sample per channel of a tensor.

Parameters

tensor – tensor with shape NCDHW

Returns

a tensor of shape NC

trw.utils.global_max_pooling_3d(tensor: trw.basic_typing.TorchTensorNCX) → trw.basic_typing.TorchTensorNCX

3D Global max pooling.

Calculate the max value per sample per channel of a tensor.

Parameters

tensor – tensor with shape NCDHW

Returns

a tensor of shape NC

trw.utils.batch_pad(array: trw.basic_typing.TensorNCX, padding: trw.basic_typing.ShapeCX, mode: str = 'edge', constant_value: trw.basic_typing.Numeric = 0)

Add padding on a numpy array of samples. This works for an arbitrary number of dimensions

Parameters

  • array – a numpy array. Samples are stored in the first dimension

  • padding – a sequence of size len(array.shape)-1 indicating the width of the padding to be added at the beginning and at the end of each dimension (except for dimension 0)

  • mode – numpy.pad mode

  • constant_value – constant used if mode == constant

Returns

a padded array

trw.utils.batch_pad_joint(arrays: List[trw.basic_typing.TensorNCX], padding: trw.basic_typing.ShapeCX, mode: str = 'edge', constant_value: trw.basic_typing.Numeric = 0)

Add padding on a list of numpy or tensor array of samples. Supports arbitrary number of dimensions

Parameters

  • arrays – a numpy array. Samples are stored in the first dimension

  • padding – a sequence of size len(array.shape)-1 indicating the width of the padding to be added at the beginning and at the end of each dimension (except for dimension 0)

  • mode – numpy.pad mode

  • constant_value – constant used if mode == constant

Returns

a list of padded arrays

trw.utils.batch_pad_torch(array: trw.basic_typing.TorchTensorNCX, padding: trw.basic_typing.ShapeCX, mode: str = 'edge', constant_value: trw.basic_typing.Numeric = 0)

Add padding on a numpy array of samples. This works for an arbitrary number of dimensions

This function mimics the API of transform_batch_pad_numpy so they can be easily interchanged.

Parameters

  • array – a Torch array. Samples are stored in the first dimension

  • padding – a sequence of size len(array.shape)-1 indicating the width of the padding to be added at the beginning and at the end of each dimension (except for dimension 0)

  • mode – numpy.pad mode. Currently supported are (‘constant’, ‘edge’, ‘symmetric’)

  • constant_value – constant used if mode == constant

Returns

a padded array

trw.utils.batch_pad_numpy(array: trw.basic_typing.NumpyTensorNCX, padding: trw.basic_typing.ShapeCX, mode: str = 'edge', constant_value: trw.basic_typing.Numeric = 0)

Add padding on a numpy array of samples. This works for an arbitrary number of dimensions

Parameters

  • array – a numpy array. Samples are stored in the first dimension

  • padding – a sequence of size len(array.shape)-1 indicating the width of the padding to be added at the beginning and at the end of each dimension (except for dimension 0)

  • mode – numpy.pad mode

  • constant_value – constant used if mode == constant

Returns

a padded array

trw.utils.batch_pad_minmax(array: trw.basic_typing.TensorNCX, padding_min: trw.basic_typing.ShapeCX, padding_max: trw.basic_typing.ShapeCX, mode: str = 'edge', constant_value: trw.basic_typing.Numeric = 0) → trw.basic_typing.TensorNCX

Add padding on a numpy array of samples. This works for an arbitrary number of dimensions

Parameters

  • array – a numpy array. Samples are stored in the first dimension

  • padding_min – a sequence of size len(array.shape)-1 indicating the width of the padding to be added at the beginning of each dimension (except for dimension 0)

  • padding_max – a sequence of size len(array.shape)-1 indicating the width of the padding to be added at the end of each dimension (except for dimension 0)

  • mode – numpy.pad mode

  • constant_value – constant used if mode == constant

Returns

a padded array

trw.utils.batch_pad_minmax_joint(arrays: List[trw.basic_typing.TensorNCX], padding_min: trw.basic_typing.ShapeCX, padding_max: trw.basic_typing.ShapeCX, mode: str = 'edge', constant_value: trw.basic_typing.Numeric = 0) → List[trw.basic_typing.TensorNCX]

Add padding on a list of numpy or tensor array of samples. Supports arbitrary number of dimensions

Parameters

  • arrays – a numpy array. Samples are stored in the first dimension

  • padding_min – a sequence of size len(array.shape)-1 indicating the width of the padding to be added at the beginning of each dimension (except for dimension 0)

  • padding_max – a sequence of size len(array.shape)-1 indicating the width of the padding to be added at the end of each dimension (except for dimension 0)

  • mode – numpy.pad mode

  • constant_value – constant used if mode == constant

Returns

a list of padded arrays

trw.utils.batch_pad_minmax_numpy(array: trw.basic_typing.NumpyTensorNCX, padding_min: trw.basic_typing.ShapeCX, padding_max: trw.basic_typing.ShapeCX, mode: str = 'edge', constant_value: trw.basic_typing.Numeric = 0) → trw.basic_typing.NumpyTensorNCX

Add padding on a numpy array of samples. This works for an arbitrary number of dimensions

Parameters

  • array – a numpy array. Samples are stored in the first dimension

  • padding_min – a sequence of size len(array.shape)-1 indicating the width of the padding to be added at the beginning of each dimension (except for dimension 0)

  • padding_max – a sequence of size len(array.shape)-1 indicating the width of the padding to be added at the end of each dimension (except for dimension 0)

  • mode – numpy.pad mode

  • constant_value – constant used if mode == constant

Returns

a padded array

trw.utils.batch_pad_minmax_torch(array: trw.basic_typing.TorchTensorNCX, padding_min: trw.basic_typing.ShapeCX, padding_max: trw.basic_typing.ShapeCX, mode: str = 'edge', constant_value: trw.basic_typing.Numeric = 0) → trw.basic_typing.TorchTensorNCX

Add padding on a numpy array of samples. This works for an arbitrary number of dimensions

This function mimics the API of transform_batch_pad_numpy so they can be easily interchanged.

Parameters

  • array – a Torch array. Samples are stored in the first dimension

  • padding_min – a sequence of size len(array.shape)-1 indicating the width of the padding to be added at the beginning of each dimension (except for dimension 0)

  • padding_max – a sequence of size len(array.shape)-1 indicating the width of the padding to be added at the end of each dimension (except for dimension 0)

  • mode – numpy.pad mode. Currently supported are (‘constant’, ‘edge’, ‘symmetric’)

  • constant_value – constant used if mode == constant

Returns

a padded array

trw.utils.safe_filename(filename: str, replace_with: str = '_') → str

Replace problematic characters (e.g., ‘/’ or ‘#’) considering Windows/Linux based OSes :param replace_with: the character to be replaced with :param filename: a filename

Returns

a string that can be used as filename

trw.utils.optional_import(module_name: str, additional_error_message='')

Optional module import.

Raise an error only when a module is being used

Parameters

  • module_name – the name of the module to import

  • additional_error_message – add a custom error message

Returns

the module

Examples

>>> nn = optional_import('torch.nn')
>>> print(nn.ReLU)
trw.utils.torch_requires(min_version: str, silent_fail: bool = False)

Requires a minimum pytorch version.

If the version is not satisfy, the requested function will throw a RuntimeError when called (but not when declared) if silent_fail is False. Otherwise, the function is not called and None is returned.

Parameters

  • min_version – a version represented as string

  • silent_fail – if True, the decorated function will not be executed and return None. If False, throw a RuntimeError

trw.utils.find_global_name(name)

Find a function or class from its name. If not found, raise a RuntimeError

Examples

>>> find_global_name('trw.utils.find_global_name')
Parameters

name – a name with possibly namespaces

Returns

an object

trw.utils.bytes2human(n: Union[int, float]) → str

Format large number of bytes into readable string for a human

Examples

>>> bytes2human(10000)
'9.8K'

>>> bytes2human(100001221)
'95.4M'
trw.utils.number2human(n: Union[int, float]) → str

Format large number into readable string for a human

Examples

>>> number2human(1000)
'1.0K'

>>> number2human(1200000)
'1.2M'
trw.utils.collect_hierarchical_module_name(base_name, model, module_to_name=None)

Create a meaningful name of the module based on the module hierarchy

Parameters

  • base_name – the base name

  • model – the model

  • module_to_name – where to store the module to name conversion

Returns

a dictionary with mapping nn.Module to string

trw.utils.collect_hierarchical_parameter_name(base_name, model, parameter_to_name=None, with_grad_only=False)

Create a meaningful name of the module’s parameters based on the module hierarchy

Parameters

  • base_name – the base name

  • model – the model

  • parameter_to_name – where to store the module to name conversion

  • with_grad_only – only the parameters requiring gradient are collected

Returns

a dictionary with mapping nn.Parameter to string

trw.utils.get_batch_n(split, nb_samples, indices, transforms, use_advanced_indexing)

Collect the split indices given and apply a series of transformations

Parameters

  • nb_samples – the total number of samples of split

  • split – a mapping of np.ndarray or torch.Tensor

  • indices – a list of indices as numpy array

  • transforms – a transformation or list of transformations or None

  • use_advanced_indexing – if True, use the advanced indexing mechanism else use a simple list (original data is referenced) advanced indexing is typically faster for small objects, however for large objects (e.g., 3D data) the advanced indexing makes a copy of the data making it very slow.

Returns

a split with the indices provided

trw.utils.to_value(v)

Convert where appropriate from tensors to numpy arrays

Parameters

v – an object. If torch.Tensor, the tensor will be converted to a numpy array. Else returns the original v

Returns

torch.Tensor as numpy arrays. Any other type will be left unchanged

trw.utils.safe_lookup(dictionary, *keys, default=None)

Recursively access nested dictionaries

Parameters

  • dictionary – nested dictionary

  • *keys – the keys to access within the nested dictionaries

  • default – the default value if dictionary is None or it doesn’t contain the keys

Returns

None if we can’t access to all the keys, else dictionary[key_0][key_1][…][key_n]

trw.utils.recursive_dict_update(dict, dict_update)
This adds any missing element from dict_update to dict, while keeping any key not

present in dict_update

Parameters

  • dict – the dictionary to be updated

  • dict_update – the updated values

trw.utils.len_batch(batch)
Parameters

batch – a data split or a collections.Sequence

Returns

the number of elements within a data split

trw.utils.flatten_nested_dictionaries(d, root_name='', delimiter='-')

Recursively flatten a dictionary of arbitrary nested size into a flattened dictionary of nested size 1

Parameters

  • d – a dictionary

  • root_name – the root name to be appended of the keys of d

  • delimiter – use this string as delimiter to concatenate nested dictionaries

Returns

a dictionary of maximum depth 1

exception trw.utils.ExceptionAbortRun(history, metrics=None, reason=None)

Bases: BaseException

The run has been pruned due to performance reason

__str__(self)

Return str(self).