CMP events

Wolfgang Roth presents Combinatorial optimization by discrete-valued neural networks and structure learning

On 2021-06-24 11:00 at
This talk covers two distinct topics that rely on the concept of performing
combinatorial optimization through continuous optimization of probability
distributions. The first topic is concerned with discrete-valued neural
networks, i.e., neural networks with discrete weights and activations. We show
how to train discrete-valued neural networks by optimizing a discrete
distribution over the weights. Once training of the distribution has finished,
a discrete-valued neural network is inferred by taking its most probable
weights or by sampling from it. The training of the distributions is performed
by minimizing an expected loss, and we present techniques to accomplish this
using stochastic gradient descent. The effectiveness of the method is
demonstrated in various experiments.
The second topic is concerned with structure learning in Bayesian networks. In
particular, we propose a differentiable approach to training tree-augmented
naive Bayes (TAN) structures based on continuous optimization of a distribution
over TAN structures. This allows us to train the parameters of a Bayesian
network and its structure jointly according to the same discriminative
criterion using gradient-based optimization. By introducing a model size
penalty to the loss function, the method can be used to trade off between model
size and accuracy. Our method consistently outperforms random TAN structures
and generative Chow-Liu TAN structures.

Recording link: