learnergy.models.gaussian

Gaussian-valued models.

A package contaning gaussian-valued models (networks) for all common learnergy modules.

class learnergy.models.gaussian.GaussianConvRBM(visible_shape: Optional[Tuple[int, int]] = (28, 28), filter_shape: Optional[Tuple[int, int]] = (7, 7), n_filters: Optional[int] = 5, n_channels: Optional[int] = 1, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.1, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, maxpooling: Optional[bool] = False, pooling_kernel: Optional[int] = 2, use_gpu: Optional[bool] = False)

Bases: learnergy.models.bernoulli.ConvRBM

A GaussianConvRBM class provides the basic implementation for Gaussian-based Convolutional Restricted Boltzmann Machines.

References

H. Lee, et al. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Proceedings of the 26th annual international conference on machine learning (2009).

__init__(self, visible_shape: Optional[Tuple[int, int]] = (28, 28), filter_shape: Optional[Tuple[int, int]] = (7, 7), n_filters: Optional[int] = 5, n_channels: Optional[int] = 1, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.1, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, maxpooling: Optional[bool] = False, pooling_kernel: Optional[int] = 2, use_gpu: Optional[bool] = False)

Initialization method.

Parameters

  • visible_shape – Shape of visible units.

  • filter_shape – Shape of filters.

  • n_filters – Number of filters.

  • n_channels – Number of channels.

  • steps – Number of Gibbs’ sampling steps.

  • learning_rate – Learning rate.

  • momentum – Momentum parameter.

  • decay – Weight decay used for penalization.

  • use_gpu – Whether GPU should be used or not.

fit(self, dataset: torch.utils.data.Dataset, batch_size: Optional[int] = 128, epochs: Optional[int] = 10)

Fits a new GaussianConvRBM model.

Parameters

  • dataset – A Dataset object containing the training data.

  • batch_size – Amount of samples per batch.

  • epochs – Number of training epochs.

Returns

MSE (mean squared error) from the training step.

Return type

(float)

hidden_sampling(self, v: torch.Tensor)

Performs the hidden layer sampling, i.e., P(h|v).

Parameters

v (torch.Tensor) – A tensor incoming from the visible layer.

Returns

The probabilities and states of the hidden layer sampling.

Return type

(Tuple[torch.Tensor, torch.Tensor])

property normalize(self)

Inner data normalization.

visible_sampling(self, h: torch.Tensor)

Performs the visible layer sampling, i.e., P(v|h).

Parameters

h – A tensor incoming from the hidden layer.

Returns

The probabilities and states of the visible layer sampling.

Return type

(Tuple[torch.Tensor, torch.Tensor])

class learnergy.models.gaussian.GaussianConvRBM4deep(visible_shape: Optional[Tuple[int, int]] = (28, 28), filter_shape: Optional[Tuple[int, int]] = (7, 7), n_filters: Optional[int] = 5, n_channels: Optional[int] = 1, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.1, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, maxpooling: Optional[bool] = False, pooling_kernel: Optional[int] = 2, use_gpu: Optional[bool] = False)

Bases: learnergy.models.bernoulli.ConvRBM

A GaussianConvRBM class provides the basic implementation for Gaussian-based Convolutional Restricted Boltzmann Machines.

References

H. Lee, et al. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Proceedings of the 26th annual international conference on machine learning (2009).

__init__(self, visible_shape: Optional[Tuple[int, int]] = (28, 28), filter_shape: Optional[Tuple[int, int]] = (7, 7), n_filters: Optional[int] = 5, n_channels: Optional[int] = 1, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.1, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, maxpooling: Optional[bool] = False, pooling_kernel: Optional[int] = 2, use_gpu: Optional[bool] = False)

Initialization method.

Parameters

  • visible_shape – Shape of visible units.

  • filter_shape – Shape of filters.

  • n_filters – Number of filters.

  • n_channels – Number of channels.

  • steps – Number of Gibbs’ sampling steps.

  • learning_rate – Learning rate.

  • momentum – Momentum parameter.

  • decay – Weight decay used for penalization.

  • use_gpu – Whether GPU should be used or not.

fit(self, dataset: torch.utils.data.Dataset, batch_size: Optional[int] = 128, epochs: Optional[int] = 10)

Fits a new GaussianConvRBM model.

Parameters

  • dataset – A Dataset object containing the training data.

  • batch_size – Amount of samples per batch.

  • epochs – Number of training epochs.

Returns

MSE (mean squared error) from the training step.

Return type

(float)

hidden_sampling(self, v: torch.Tensor)

Performs the hidden layer sampling, i.e., P(h|v).

Parameters

v (torch.Tensor) – A tensor incoming from the visible layer.

Returns

The probabilities and states of the hidden layer sampling.

Return type

(Tuple[torch.Tensor, torch.Tensor])

property normalize(self)

Inner data normalization.

visible_sampling(self, h: torch.Tensor)

Performs the visible layer sampling, i.e., P(v|h).

Parameters

h – A tensor incoming from the hidden layer.

Returns

The probabilities and states of the visible layer sampling.

Return type

(Tuple[torch.Tensor, torch.Tensor])

class learnergy.models.gaussian.GaussianRBM(n_visible: Optional[int] = 128, n_hidden: Optional[int] = 128, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.1, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, temperature: Optional[float] = 1.0, use_gpu: Optional[bool] = False, normalize: Optional[bool] = True, input_normalize: Optional[bool] = True)

Bases: learnergy.models.bernoulli.RBM

A GaussianRBM class provides the basic implementation for Gaussian-Bernoulli Restricted Boltzmann Machines (with standardization).

Note that this classes normalize the data as it uses variance equals to one throughout its learning procedure.

This is a trick to ease the calculations of the hidden and visible layer samplings, as well as the cost function.

References

K. Cho, A. Ilin, T. Raiko. Improved learning of Gaussian-Bernoulli restricted Boltzmann machines. International conference on artificial neural networks (2011).

__init__(self, n_visible: Optional[int] = 128, n_hidden: Optional[int] = 128, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.1, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, temperature: Optional[float] = 1.0, use_gpu: Optional[bool] = False, normalize: Optional[bool] = True, input_normalize: Optional[bool] = True)

Initialization method.

Parameters

  • n_visible – Amount of visible units.

  • n_hidden – Amount of hidden units.

  • steps – Number of Gibbs’ sampling steps.

  • learning_rate – Learning rate.

  • momentum – Momentum parameter.

  • decay – Weight decay used for penalization.

  • temperature – Temperature factor.

  • use_gpu – Whether GPU should be used or not.

  • normalize – Whether or not to use batch normalization.

  • input_normalize – Whether or not to normalize inputs.

energy(self, samples: torch.Tensor)

Calculates and frees the system’s energy.

Parameters

samples – Samples to be energy-freed.

Returns

The system’s energy based on input samples.

Return type

(torch.Tensor)

fit(self, dataset: torch.utils.data.Dataset, batch_size: Optional[int] = 128, epochs: Optional[int] = 10)

Fits a new GaussianRBM model.

Parameters

  • dataset – A Dataset object containing the training data.

  • batch_size – Amount of samples per batch.

  • epochs – Number of training epochs.

Returns

MSE (mean squared error) and log pseudo-likelihood from the training step.

Return type

(Tuple[float, float])

forward(self, x: torch.Tensor)

Performs a forward pass over the data.

Parameters

x – An input tensor for computing the forward pass.

Returns

A tensor containing the RBM’s outputs.

Return type

(torch.Tensor)

property input_normalize(self)

Whether or not to use input normalization.

property normalize(self)

Whether or not to use batch normalization.

reconstruct(self, dataset: torch.utils.data.Dataset)

Reconstructs batches of new samples.

Parameters

dataset (torch.utils.data.Dataset) – A Dataset object containing the testing data.

Returns

Reconstruction error and visible probabilities, i.e., P(v|h).

Return type

(Tuple[float, torch.Tensor])

visible_sampling(self, h: torch.Tensor, scale: Optional[bool] = False)

Performs the visible layer sampling, i.e., P(v|h).

Parameters

  • h – A tensor incoming from the hidden layer.

  • scale – A boolean to decide whether temperature should be used or not.

Returns

The probabilities and states of the visible layer sampling.

Return type

(Tuple[torch.Tensor, torch.Tensor])

class learnergy.models.gaussian.GaussianRBM4deep(n_visible: Optional[int] = 128, n_hidden: Optional[int] = 128, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.1, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, temperature: Optional[float] = 1.0, use_gpu: Optional[bool] = False, normalize: Optional[bool] = True, input_normalize: Optional[bool] = True)

Bases: GaussianRBM

A GaussianRBM class provides the basic implementation for Gaussian-Bernoulli Restricted Boltzmann Machines (with standardization).

Note that this classes normalize the data as it uses variance equals to one throughout its learning procedure.

This is a trick to ease the calculations of the hidden and visible layer samplings, as well as the cost function.

References

K. Cho, A. Ilin, T. Raiko. Improved learning of Gaussian-Bernoulli restricted Boltzmann machines. International conference on artificial neural networks (2011).

__init__(self, n_visible: Optional[int] = 128, n_hidden: Optional[int] = 128, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.1, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, temperature: Optional[float] = 1.0, use_gpu: Optional[bool] = False, normalize: Optional[bool] = True, input_normalize: Optional[bool] = True)

Initialization method.

Parameters

  • n_visible – Amount of visible units.

  • n_hidden – Amount of hidden units.

  • steps – Number of Gibbs’ sampling steps.

  • learning_rate – Learning rate.

  • momentum – Momentum parameter.

  • decay – Weight decay used for penalization.

  • temperature – Temperature factor.

  • use_gpu – Whether GPU should be used or not.

  • normalize – Whether or not to use batch normalization.

  • input_normalize – Whether or not to normalize inputs.

fit(self, dataset: torch.utils.data.Dataset, batch_size: Optional[int] = 128, epochs: Optional[int] = 1)

Fits a new GaussianRBM model.

Parameters

  • dataset – A Dataset object containing the training data.

  • batch_size – Amount of samples per batch.

  • epochs – Number of training epochs.

Returns

MSE (mean squared error) and log pseudo-likelihood from the training step.

Return type

(Tuple[float, float])

class learnergy.models.gaussian.GaussianReluRBM(n_visible: Optional[int] = 128, n_hidden: Optional[int] = 128, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.001, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, temperature: Optional[float] = 1.0, use_gpu: Optional[bool] = False, normalize: Optional[bool] = True, input_normalize: Optional[bool] = True)

Bases: GaussianRBM

A GaussianReluRBM class provides the basic implementation for Gaussian-ReLU Restricted Boltzmann Machines (for raw pixels values).

Note that this class requires raw data (integer-valued) in order to model the image covariance into a latent ReLU layer.

References

G. Hinton. A practical guide to training restricted Boltzmann machines. Neural networks: Tricks of the trade (2012).

__init__(self, n_visible: Optional[int] = 128, n_hidden: Optional[int] = 128, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.001, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, temperature: Optional[float] = 1.0, use_gpu: Optional[bool] = False, normalize: Optional[bool] = True, input_normalize: Optional[bool] = True)

Initialization method.

Parameters

  • n_visible – Amount of visible units.

  • n_hidden – Amount of hidden units.

  • steps – Number of Gibbs’ sampling steps.

  • learning_rate – Learning rate.

  • momentum – Momentum parameter.

  • decay – Weight decay used for penalization.

  • temperature – Temperature factor.

  • use_gpu – Whether GPU should be used or not.

  • normalize – Whether or not to use batch normalization.

  • input_normalize – Whether or not to normalize inputs.

hidden_sampling(self, v: torch.Tensor, scale: Optional[bool] = False)

Performs the hidden layer sampling, i.e., P(h|v).

Parameters

  • v – A tensor incoming from the visible layer.

  • scale – A boolean to decide whether temperature should be used or not.

Returns

The probabilities and states of the hidden layer sampling.

Return type

(Tuple[torch.Tensor, torch.Tensor])

class learnergy.models.gaussian.GaussianReluRBM4deep(n_visible: Optional[int] = 128, n_hidden: Optional[int] = 128, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.001, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, temperature: Optional[float] = 1.0, use_gpu: Optional[bool] = False, normalize: Optional[bool] = True, input_normalize: Optional[bool] = True)

Bases: GaussianRBM4deep

A GaussianReluRBM class provides the basic implementation for Gaussian-ReLU Restricted Boltzmann Machines (for raw pixels values).

Note that this class requires raw data (integer-valued) in order to model the image covariance into a latent ReLU layer.

References

G. Hinton. A practical guide to training restricted Boltzmann machines. Neural networks: Tricks of the trade (2012).

__init__(self, n_visible: Optional[int] = 128, n_hidden: Optional[int] = 128, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.001, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, temperature: Optional[float] = 1.0, use_gpu: Optional[bool] = False, normalize: Optional[bool] = True, input_normalize: Optional[bool] = True)

Initialization method.

Parameters

  • n_visible – Amount of visible units.

  • n_hidden – Amount of hidden units.

  • steps – Number of Gibbs’ sampling steps.

  • learning_rate – Learning rate.

  • momentum – Momentum parameter.

  • decay – Weight decay used for penalization.

  • temperature – Temperature factor.

  • use_gpu – Whether GPU should be used or not.

  • normalize – Whether or not to use batch normalization.

  • input_normalize – Whether or not to normalize inputs.

hidden_sampling(self, v: torch.Tensor, scale: Optional[bool] = False)

Performs the hidden layer sampling, i.e., P(h|v).

Parameters

  • v – A tensor incoming from the visible layer.

  • scale – A boolean to decide whether temperature should be used or not.

Returns

The probabilities and states of the hidden layer sampling.

Return type

(Tuple[torch.Tensor, torch.Tensor])

class learnergy.models.gaussian.GaussianSeluRBM(n_visible: Optional[int] = 128, n_hidden: Optional[int] = 128, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.001, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, temperature: Optional[float] = 1.0, use_gpu: Optional[bool] = False, normalize: Optional[bool] = False, input_normalize: Optional[bool] = True)

Bases: GaussianRBM

A GaussianSeluRBM class provides the basic implementation for Gaussian-SeLU Restricted Boltzmann Machines (for raw pixels values).

Note that this class requires raw data (integer-valued) in order to model the image covariance into a latent ReLU layer.

References

G. Hinton. A practical guide to training restricted Boltzmann machines. Neural networks: Tricks of the trade (2012).

G. Klambauer et al. Self-normalizing neural networks. Proceedings, NIPS (2017).

__init__(self, n_visible: Optional[int] = 128, n_hidden: Optional[int] = 128, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.001, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, temperature: Optional[float] = 1.0, use_gpu: Optional[bool] = False, normalize: Optional[bool] = False, input_normalize: Optional[bool] = True)

Initialization method.

Parameters

  • n_visible – Amount of visible units.

  • n_hidden – Amount of hidden units.

  • steps – Number of Gibbs’ sampling steps.

  • learning_rate – Learning rate.

  • momentum – Momentum parameter.

  • decay – Weight decay used for penalization.

  • temperature – Temperature factor.

  • use_gpu – Whether GPU should be used or not.

  • normalize – Whether or not to use batch normalization.

  • input_normalize – Whether or not to normalize inputs.

hidden_sampling(self, v: torch.Tensor, scale: Optional[bool] = False)

Performs the hidden layer sampling, i.e., P(h|v).

Parameters

  • v – A tensor incoming from the visible layer.

  • scale – A boolean to decide whether temperature should be used or not.

Returns

The probabilities and states of the hidden layer sampling.

Return type

(Tuple[torch.Tensor, torch.Tensor])

class learnergy.models.gaussian.VarianceGaussianRBM(n_visible: Optional[int] = 128, n_hidden: Optional[int] = 128, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.1, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, temperature: Optional[float] = 1.0, use_gpu: Optional[bool] = False)

Bases: learnergy.models.bernoulli.RBM

A VarianceGaussianRBM class provides the basic implementation for Gaussian-Bernoulli Restricted Boltzmann Machines (without standardization).

Note that this class implements a new cost function that takes in account a new learning parameter: variance (sigma).

Therefore, there is no need to standardize the data, as the variance will be trained throughout the learning procedure.

References

K. Cho, A. Ilin, T. Raiko. Improved learning of Gaussian-Bernoulli restricted Boltzmann machines. International conference on artificial neural networks (2011).

__init__(self, n_visible: Optional[int] = 128, n_hidden: Optional[int] = 128, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.1, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, temperature: Optional[float] = 1.0, use_gpu: Optional[bool] = False)

Initialization method.

Parameters

  • n_visible – Amount of visible units.

  • n_hidden – Amount of hidden units.

  • steps – Number of Gibbs’ sampling steps.

  • learning_rate – Learning rate.

  • momentum – Momentum parameter.

  • decay – Weight decay used for penalization.

  • temperature – Temperature factor.

  • use_gpu – Whether GPU should be used or not.

energy(self, samples: torch.Tensor)

Calculates and frees the system’s energy.

Parameters

samples – Samples to be energy-freed.

Returns

The system’s energy based on input samples.

Return type

(torch.Tensor)

hidden_sampling(self, v: torch.Tensor, scale: Optional[bool] = False)

Performs the hidden layer sampling, i.e., P(h|v).

Parameters

  • v – A tensor incoming from the visible layer.

  • scale – A boolean to decide whether temperature should be used or not.

Returns

The probabilities and states of the hidden layer sampling.

Return type

(Tuple[torch.Tensor, torch.Tensor])

property sigma(self)

torch.nn.Parameter: Variance parameter.

visible_sampling(self, h: torch.Tensor, scale: Optional[bool] = False)

Performs the visible layer sampling, i.e., P(v|h).

Parameters

  • h – A tensor incoming from the hidden layer.

  • scale – A boolean to decide whether temperature should be used or not.

Returns

The probabilities and states of the visible layer sampling.

Return type

(Tuple[torch.Tensor, torch.Tensor])