learnergy.models.bernoulli

Bernoulli-based models.

A package contaning bernoulli-based models (networks) for all common learnergy modules.

class learnergy.models.bernoulli.ConvRBM(visible_shape: Optional[Tuple[int, int]] = (28, 28), filter_shape: Optional[Tuple[int, int]] = (7, 7), n_filters: Optional[int] = 5, n_channels: Optional[int] = 1, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.1, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, maxpooling: Optional[bool] = False, pooling_kernel: Optional[int] = 2, use_gpu: Optional[bool] = False)

Bases: learnergy.core.Model

A ConvRBM class provides the basic implementation for Convolutional Bernoulli-Bernoulli Restricted Boltzmann Machines.

References

H. Lee, et al. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Proceedings of the 26th annual international conference on machine learning (2009).

property W(self)

Filters’ matrix.

__init__(self, visible_shape: Optional[Tuple[int, int]] = (28, 28), filter_shape: Optional[Tuple[int, int]] = (7, 7), n_filters: Optional[int] = 5, n_channels: Optional[int] = 1, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.1, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, maxpooling: Optional[bool] = False, pooling_kernel: Optional[int] = 2, use_gpu: Optional[bool] = False)

Initialization method.

Parameters

  • visible_shape – Shape of visible units.

  • filter_shape – Shape of filters.

  • n_filters – Number of filters.

  • n_channels – Number of channels.

  • steps – Number of Gibbs’ sampling steps.

  • learning_rate – Learning rate.

  • momentum – Momentum parameter.

  • decay – Weight decay used for penalization.

  • maxpooling – Whether MaxPooling2D should be used or not.

  • pooling_kernel – The kernel size of MaxPooling layer (when maxpooling=True).

  • use_gpu – Whether GPU should be used or not.

property a(self)

Visible units bias.

property b(self)

Hidden units bias.

property decay(self)

Weight decay.

energy(self, samples: torch.Tensor)

Calculates and frees the system’s energy.

Parameters

samples – Samples to be energy-freed.

Returns

The system’s energy based on input samples.

Return type

(torch.Tensor)

property filter_shape(self)

Shape of filters.

fit(self, dataset: torch.utils.data.Dataset, batch_size: Optional[int] = 128, epochs: Optional[int] = 10)

Fits a new ConvRBM model.

Parameters

  • dataset – A Dataset object containing the training data.

  • batch_size – Amount of samples per batch.

  • epochs – Number of training epochs.

Returns

MSE (mean squared error) from the training step.

Return type

(float)

forward(self, x: torch.Tensor)

Performs a forward pass over the data.

Parameters

x – An input tensor for computing the forward pass.

Returns

A tensor containing the Convolutional RBM’s outputs.

Return type

(torch.Tensor)

gibbs_sampling(self, v: torch.Tensor)

Performs the whole Gibbs sampling procedure.

Parameters

v – A tensor incoming from the visible layer.

Returns

The probabilities and states of the hidden layer sampling (positive),

the probabilities and states of the hidden layer sampling (negative) and the states of the visible layer sampling (negative).

Return type

(Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor])

hidden_sampling(self, v: torch.Tensor)

Performs the hidden layer sampling, i.e., P(h|v).

Parameters

v – A tensor incoming from the visible layer.

Returns

The probabilities and states of the hidden layer sampling.

Return type

(Tuple[torch.Tensor, torch.Tensor])

property hidden_shape(self)

Shape of hidden units.

property lr(self)

Learning rate.

property maxpooling(self)

Usage of MaxPooling.

property momentum(self)

Momentum parameter.

property n_channels(self)

Number of channels.

property n_filters(self)

Number of filters.

property optimizer(self)

Stochastic Gradient Descent object.

reconstruct(self, dataset: torch.utils.data.Dataset)

Reconstructs batches of new samples.

Parameters

dataset – A Dataset object containing the testing data.

Returns

Reconstruction error and visible probabilities, i.e., P(v|h).

Return type

(Tuple[float, torch.Tensor])

property steps(self)

Number of steps Gibbs’ sampling steps.

visible_sampling(self, h: torch.Tensor)

Performs the visible layer sampling, i.e., P(v|h).

Parameters

h – A tensor incoming from the hidden layer.

Returns

The probabilities and states of the visible layer sampling.

Return type

(Tuple[torch.Tensor, torch.Tensor])

property visible_shape(self)

Shape of visible units.

class learnergy.models.bernoulli.DiscriminativeRBM(n_visible: Optional[int] = 128, n_hidden: Optional[int] = 128, n_classes: Optional[int] = 1, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.1, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, temperature: Optional[float] = 1.0, use_gpu: Optional[bool] = False)

Bases: learnergy.models.bernoulli.RBM

A DiscriminativeRBM class provides the basic implementation for Discriminative Bernoulli-Bernoulli Restricted Boltzmann Machines.

References

H. Larochelle and Y. Bengio. Classification using discriminative restricted Boltzmann machines. Proceedings of the 25th international conference on Machine learning (2008).

property U(self)

Class weights’ matrix.

__init__(self, n_visible: Optional[int] = 128, n_hidden: Optional[int] = 128, n_classes: Optional[int] = 1, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.1, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, temperature: Optional[float] = 1.0, use_gpu: Optional[bool] = False)

Initialization method.

Parameters

  • n_visible – Amount of visible units.

  • n_hidden – Amount of hidden units.

  • n_classes – Amount of classes.

  • steps – Number of Gibbs’ sampling steps.

  • learning_rate – Learning rate.

  • momentum – Momentum parameter.

  • decay – Weight decay used for penalization.

  • temperature – Temperature factor.

  • use_gpu – Whether GPU should be used or not.

property c(self)

Class units bias.

fit(self, dataset: torch.utils.data.Dataset, batch_size: Optional[int] = 128, epochs: Optional[int] = 10)

Fits a new DRBM model.

Parameters

  • dataset – A Dataset object containing the training data.

  • batch_size – Amount of samples per batch.

  • epochs – Number of training epochs.

Returns

Loss and accuracy from the training step.

Return type

(Tuple[float, float])

labels_sampling(self, samples: torch.Tensor)

Calculates labels probabilities by samplings, i.e., P(y|v).

Parameters

samples – Samples to be labels-calculated.

Returns

Labels’ probabilities based on input samples.

Return type

(torch.Tensor)

property loss(self)

Cross-Entropy loss function.

property n_classes(self)

Number of classes.

predict(self, dataset: torch.utils.data.Dataset)

Predicts batches of new samples.

Parameters

dataset – A Dataset object containing the testing data.

Returns

Accuracy, prediction probabilities and labels, i.e., P(y|v).

Return type

(Tuple[float, torch.Tensor, torch.Tensor])

class learnergy.models.bernoulli.DropConnectRBM(n_visible: Optional[int] = 128, n_hidden: Optional[int] = 128, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.1, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, temperature: Optional[float] = 1.0, dropout: Optional[float] = 0.5, use_gpu: Optional[bool] = False)

Bases: DropoutRBM

A DropConnectRBM class provides the basic implementation for Bernoulli-Bernoulli Restricted Boltzmann Machines along with a DropConnect regularization.

References

N. Srivastava, et al. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research (2014).

__init__(self, n_visible: Optional[int] = 128, n_hidden: Optional[int] = 128, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.1, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, temperature: Optional[float] = 1.0, dropout: Optional[float] = 0.5, use_gpu: Optional[bool] = False)

Initialization method.

Parameters

  • n_visible – Amount of visible units.

  • n_hidden – Amount of hidden units.

  • steps – Number of Gibbs’ sampling steps.

  • learning_rate – Learning rate.

  • momentum – Momentum parameter.

  • decay – Weight decay used for penalization.

  • temperature – Temperature factor.

  • dropout – Dropout rate.

  • use_gpu – Whether GPU should be used or not.

hidden_sampling(self, v: torch.Tensor, scale: Optional[bool] = False)

Performs the hidden layer sampling using a dropconnect mask, i.e., P(h|m,v).

Parameters

  • v – A tensor incoming from the visible layer.

  • scale – A boolean to decide whether temperature should be used or not.

Returns

The probabilities and states of the hidden layer sampling.

Return type

(Tuple[torch.Tensor, torch.Tensor])

class learnergy.models.bernoulli.DropoutRBM(n_visible: Optional[int] = 128, n_hidden: Optional[int] = 128, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.1, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, temperature: Optional[float] = 1.0, dropout: Optional[float] = 0.5, use_gpu: Optional[bool] = False)

Bases: learnergy.models.bernoulli.RBM

A DropoutRBM class provides the basic implementation for Bernoulli-Bernoulli Restricted Boltzmann Machines along with a Dropout regularization.

References

N. Srivastava, et al. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research (2014).

__init__(self, n_visible: Optional[int] = 128, n_hidden: Optional[int] = 128, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.1, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, temperature: Optional[float] = 1.0, dropout: Optional[float] = 0.5, use_gpu: Optional[bool] = False)

Initialization method.

Parameters

  • n_visible – Amount of visible units.

  • n_hidden – Amount of hidden units.

  • steps – Number of Gibbs’ sampling steps.

  • learning_rate – Learning rate.

  • momentum – Momentum parameter.

  • decay – Weight decay used for penalization.

  • temperature – Temperature factor.

  • dropout – Dropout rate.

  • use_gpu – Whether GPU should be used or not.

hidden_sampling(self, v: torch.Tensor, scale: Optional[bool] = False)

Performs the hidden layer sampling using a dropout mask, i.e., P(h|r,v).

Parameters

  • v – A tensor incoming from the visible layer.

  • scale – A boolean to decide whether temperature should be used or not.

Returns

The probabilities and states of the hidden layer sampling.

Return type

(Tuple[torch.Tensor, torch.Tensor])

property p(self)

Probability of applying dropout.

reconstruct(self, dataset: torch.utils.data.Dataset)

Reconstructs batches of new samples.

Parameters

dataset – A Dataset object containing the testing data.

Returns

Reconstruction error and visible probabilities, i.e., P(v|h).

Return type

(Tuple[float, torch.Tensor])

class learnergy.models.bernoulli.HybridDiscriminativeRBM(n_visible: Optional[int] = 128, n_hidden: Optional[int] = 128, n_classes: Optional[int] = 1, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.1, alpha: Optional[float] = 0.01, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, temperature: Optional[float] = 1.0, use_gpu: Optional[bool] = False)

Bases: DiscriminativeRBM

A HybridDiscriminativeRBM class provides the basic implementation for Hybrid Discriminative Bernoulli-Bernoulli Restricted Boltzmann Machines.

References

H. Larochelle and Y. Bengio. Classification using discriminative restricted Boltzmann machines. Proceedings of the 25th international conference on Machine learning (2008).

__init__(self, n_visible: Optional[int] = 128, n_hidden: Optional[int] = 128, n_classes: Optional[int] = 1, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.1, alpha: Optional[float] = 0.01, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, temperature: Optional[float] = 1.0, use_gpu: Optional[bool] = False)

Initialization method.

Parameters

  • n_visible – Amount of visible units.

  • n_hidden – Amount of hidden units.

  • n_classes – Amount of classes.

  • steps – Number of Gibbs’ sampling steps.

  • learning_rate – Learning rate.

  • alpha – Amount of penalization to the generative loss.

  • momentum – Momentum parameter.

  • decay – Weight decay used for penalization.

  • temperature – Temperature factor.

  • use_gpu – Whether GPU should be used or not.

property alpha(self)

Generative loss penalization.

class_sampling(self, h: torch.Tensor)

Performs the class layer sampling, i.e., P(y|h).

Parameters

h – A tensor incoming from the hidden layer.

Returns

The probabilities and states of the class layer sampling.

Return type

(Tuple[torch.Tensor, torch.Tensor])

fit(self, dataset: torch.utils.data.Dataset, batch_size: Optional[int] = 128, epochs: Optional[int] = 10)

Fits a new DRBM model.

Parameters

  • dataset – A Dataset object containing the training data.

  • batch_size – Amount of samples per batch.

  • epochs – Number of training epochs.

Returns

Loss and accuracy from the training step.

Return type

(Tuple[float, float])

gibbs_sampling(self, v: torch.Tensor, y: torch.Tensor)

Performs the whole Gibbs sampling procedure.

Parameters

  • v – A tensor incoming from the visible layer.

  • y – A tensor incoming from the class layer.

Returns

The probabilities and states of the hidden layer sampling (positive),

the probabilities and states of the hidden layer sampling (negative) and the states of the visible layer sampling (negative).

Return type

(Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor])

hidden_sampling(self, v: torch.Tensor, y: torch.Tensor, scale: Optional[bool] = False)

Performs the hidden layer sampling, i.e., P(h|y,v).

Parameters

  • v – A tensor incoming from the visible layer.

  • y – A tensor incoming from the class layer.

  • scale – A boolean to decide whether temperature should be used or not.

Returns

The probabilities and states of the hidden layer sampling.

Return type

(Tuple[torch.Tensor, torch.Tensor])

class learnergy.models.bernoulli.RBM(n_visible: Optional[int] = 128, n_hidden: Optional[int] = 128, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.1, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, temperature: Optional[float] = 1.0, use_gpu: Optional[bool] = False)

Bases: learnergy.core.Model

An RBM class provides the basic implementation for Bernoulli-Bernoulli Restricted Boltzmann Machines.

References

G. Hinton. A practical guide to training restricted Boltzmann machines. Neural networks: Tricks of the trade (2012).

property T(self)

Temperature factor.

property W(self)

Weights’ matrix.

__init__(self, n_visible: Optional[int] = 128, n_hidden: Optional[int] = 128, steps: Optional[int] = 1, learning_rate: Optional[float] = 0.1, momentum: Optional[float] = 0.0, decay: Optional[float] = 0.0, temperature: Optional[float] = 1.0, use_gpu: Optional[bool] = False)

Initialization method.

Parameters

  • n_visible – Amount of visible units.

  • n_hidden – Amount of hidden units.

  • steps – Number of Gibbs’ sampling steps.

  • learning_rate – Learning rate.

  • momentum – Momentum parameter.

  • decay – Weight decay used for penalization.

  • temperature – Temperature factor.

  • use_gpu – Whether GPU should be used or not.

property a(self)

Visible units bias.

property b(self)

Hidden units bias.

property decay(self)

Weight decay.

energy(self, samples: torch.Tensor)

Calculates and frees the system’s energy.

Parameters

samples – Samples to be energy-freed.

Returns

The system’s energy based on input samples.

Return type

(torch.Tensor)

fit(self, dataset: torch.utils.data.Dataset, batch_size: Optional[int] = 128, epochs: Optional[int] = 10)

Fits a new RBM model.

Parameters

  • dataset – A Dataset object containing the training data.

  • batch_size – Amount of samples per batch.

  • epochs – Number of training epochs.

Returns

MSE (mean squared error) and log pseudo-likelihood from the training step.

Return type

(Tuple[float, float])

forward(self, x: torch.Tensor)

Performs a forward pass over the data.

Parameters

x – An input tensor for computing the forward pass.

Returns

A tensor containing the RBM’s outputs.

Return type

(torch.Tensor)

gibbs_sampling(self, v: torch.Tensor)

Performs the whole Gibbs sampling procedure.

Parameters

v – A tensor incoming from the visible layer.

Returns

The probabilities and states of the hidden layer sampling (positive),

the probabilities and states of the hidden layer sampling (negative) and the states of the visible layer sampling (negative).

Return type

(Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor])

hidden_sampling(self, v: torch.Tensor, scale: Optional[bool] = False)

Performs the hidden layer sampling, i.e., P(h|v).

Parameters

  • v – A tensor incoming from the visible layer.

  • scale – A boolean to decide whether temperature should be used or not.

Returns

The probabilities and states of the hidden layer sampling.

Return type

(torch.Tensor)

property lr(self)

Learning rate.

property momentum(self)

Momentum parameter.

property n_hidden(self)

Number of hidden units.

property n_visible(self)

Number of visible units.

property optimizer(self)

Stochastic Gradient Descent object.

pre_activation(self, v: torch.Tensor, scale: Optional[bool] = False)

Performs the pre-activation over hidden neurons, i.e., Wx’ + b.

Parameters

  • v – A tensor incoming from the visible layer.

  • scale – A boolean to decide whether temperature should be used or not.

Returns

An input for any type of activation function.

Return type

(torch.Tensor)

pseudo_likelihood(self, samples: torch.Tensor)

Calculates the logarithm of the pseudo-likelihood.

Parameters

samples – Samples to be calculated.

Returns

The logarithm of the pseudo-likelihood based on input samples.

Return type

(torch.Tensor)

reconstruct(self, dataset: torch.utils.data.Dataset)

Reconstructs batches of new samples.

Parameters

dataset – A Dataset object containing the testing data.

Returns

Reconstruction error and visible probabilities, i.e., P(v|h).

Return type

(Tuple[float, torch.Tensor])

property steps(self)

Number of steps Gibbs’ sampling steps.

visible_sampling(self, h: torch.Tensor, scale: Optional[bool] = False)

Performs the visible layer sampling, i.e., P(v|h).

Parameters

  • h – A tensor incoming from the hidden layer.

  • scale – A boolean to decide whether temperature should be used or not.

Returns

The probabilities and states of the visible layer sampling.

Return type

(torch.Tensor)