Seminar Informatik Sommer 2019!

General overview of machine and deep learning: history, models, alogrithms and applications

[1] LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. "Deep learning." nature 521.7553 (2015): 436.

[2] Simeone, Osvaldo. "A brief introduction to machine learning for engineers." Foundations and Trends® in Signal Processing 12.3-4 (2018): 200-431.

[3] Schmidhuber, Jürgen. "Deep learning in neural networks: An overview." Neural networks 61 (2015): 85-117.

Generative adversarial networks (GANs) are based on a game theoretic scenario in which the generator network must compete against an adversary. The generative network (G) produces samples of a distribution, while its adversary, the discriminator network (D), attempts to distinguish between samples drawn from the training data and samples drawn from the generator. The discriminator emits a probability value indicating the probability that a sample came from the training data, rather than from the model G. Universal adversarial perturbations pose a serious threat to the success of deep learning in practice. Adding these perturbations, single vectors of a small norm, to images in a dataset with high probability leads to misclassifying these images.

Papers
[1] Goodfellow, Ian, et al. "Generative adversarial nets." Advances in neural information processing systems. 2014.
[2] Khrulkov, Valentin, and Ivan Oseledets. "Art of singular vectors and universal adversarial perturbations." arXiv preprint
arXiv:1709.03582(2017).

The deeply complex hierarchical structure of neurons and connections in the brain play a major role in this process of remembering and labelling objects. Convolutional neural network (CNN, or ConvNet) is a class of deep, feed-forward artificial neural networks. The use of a variation of MultiLayer Perceptrons (MLP) were inspired from models of brain functionality of perceiving the world visually and enables the need for relatively little pre-processing for feature extraction. This property makes it ideal for image and video classification or to natural languate processing.

[1 ] Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. "Deep learning (adaptive computation and machine learning series)." Adaptive Computation and Machine Learning series (2016): 800. - Chapter 9.

[2] Wiatowski, Thomas, and Helmut Bölcskei. "A mathematical theory of deep convolutional neural networks for feature extraction." IEEE Transactions on Information Theory 64.3 (2018): 1845-1866.

Neural Networks principal element is the mechanism to classify incoming information as useful or less-useful. They help the network use the useful information and suppress the irrelevant data points. This seminar topic is on the brief overview of what an activation function is, about the different type of available activation functions and about how to choose which activation function to use.

Deep learning techniques have been applied with great success to many problems in machine learning. The objective of a deep learning model is to predict the output of a considered system. The parameters of such model are assigned through a learning process based on available data. In some applications, the reliability of a deep learning model, i.e. model uncertainty due to limited data or data error, is also required, e.g. under the context of Artificial Intelligence (AI) safety. Bayesian probability theory is a mathematical tool to reason about uncertainty. Applying Bayesian inference to deep learning techniques, one can build a model that can predict in a probabilistic way the system's outputs. In this seminar work, student will investigate the theory of Bayesian deep neural networks, and the inference techniques in these.

[1] Gal, Y., & Ghahramani, Z. (2016, June). Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning (pp. 1050-1059). [2] Gal, Yarin. "Uncertainty in deep learning." University of Cambridge (2016).

This seminar topic reviews numerical optimization algorithms in the context of maching learning applications.

[1] Bottou, Léon, Frank E. Curtis, and Jorge Nocedal. "Optimization methods for large-scale machine learning." SIAM Review 60.2 (2018): 223-311.

[2] Jordan, Michael I. "Dynamical, Symplectic and Stochastic Perspectives on Gradient-Based Optimization." University of California, Berkeley(2018).