Computer Science

EMANUELE RODOLA'

General goals:
Familiarity with advanced machine learning techniques, both supervised and unsupervised; modeling skills of complex problems using deep learning techniques, and their application to diverse applicative settings.

Specific goals:
Topics include: deep neural networks, their training and the interpretation of results; convolutional networks and prominent architectures; theory of deep learning and convergence; programming frameworks for implementing advanced machine learning techniques; autoencoders; adversarial attacks.

Knowledge and understanding:
How neural networks work and their mathematical interpretation as universal approximators. Understanding the limits and potentials of advanced machine learning models.

Applying knowledge and understanding:
Design, implementation, deployment and analysis of deep learning architectures addressing complex problems in several applicative areas.

Critical and judgmental abilities:
To be able to evaluate the performance of different architectures, and to assess their generalization capabilities.

Communication skills:
To be able to communicate clearly how to formulate an advanced machine learning problem as well as its implementation, its applicability in realistic settings, and specific architectural and regularization choices.

Ability to learn:
Understanding alternative and more complex techniques such as generative models based on optimal transportation, scattering transforms and the energetic profile of neural networks. To be able to implement existing techniques efficiently, robustly and reliably.

General Objectives
Gain familiarity with advanced techniques in supervised and unsupervised machine learning; acquire skills in modeling complex problems using deep learning techniques, and apply them in various application contexts.

Specific Objectives
Topics include: deep neural networks, their training and the interpretation of results; convolutional networks and prominent architectures; deep learning theory with particular reference to convergence issues; use of existing frameworks for implementing advanced machine learning techniques; autoencoders; adversarial attacks.

Knowledge and Understanding
Knowledge about the functioning of neural networks and their mathematical interpretation as universal approximators. Understanding of the limitations and potential of advanced machine learning models.

Application of Knowledge and Understanding
Design, implementation, commissioning, and analysis of deep learning architectures to solve complex problems in various application areas.

Autonomy of Judgment
Being able to evaluate the performance of different architectures, and assess their generalization capability.

Communication Skills
Being able to clearly communicate the formulation of an advanced learning problem and its implementation, its applicability in realistic contexts, as well as motivate architectural choices and regularization.

Capability for Further Learning
Being able to learn alternative and more complex techniques such as generative models based on optimal transport, scattering transforms, and the study of the energy profile of neural networks. Being able to efficiently, robustly, and reliably implement existing techniques.

Important: Calculus; linear algebra; fundamentals of machine learning.
Mandatory: Fundamentals of programming (Python language).

The course will cover the basics of calculus, linear algebra and machine learning that are needed to fully understand the lectures.

- Data, features, and embeddings
- Data awareness
- Modeling prior knowledge
- The curse of dimensionality
- Task-driven features and invariances
- Recap of linear algebra
- Vector spaces, bases
- Linear maps
- Matrix notation and matrix algebra
- Tensors and tensor operations
- Parametric models and regression
- Linear and polynomial regression
- Convexity and Lp norms
- Underfitting and overfitting
- Cross validation
- Logistic regression
- Optimization
- Gradient descent
- Stochastic gradient descent
- Learning rate, decay, momentum, batch size
- Forward and reverse-mode automatic differentiation
- Deep neural networks
- Multi layer perceptron
- Backpropagation
- Universal approximation theorems
- Autograd and modules
- Invariance, equivariance, compositionality
- Convolutional neural networks
- Pooling
- Double descent
- Regularization: weight penalty, early stopping, dropout, batchnorm
- Generative models
- PCA
- Manifolds and the manifold hypothesis
- Representation learning
- Autoencoders: variational, contractive, denoising
- Generative adversarial networks
- Adversarial learning
- Decision boundaries
- Black-box and white-box attacks
- Adversarial perturbations: universal and one-pixel
- Adversarial training
- Geometric deep learning
- Learning on graphs and point clouds
- Learning on surfaces
- Generative models of structured data
- Adversarial surfaces

Due to the highly dynamic nature of this advanced course, classes will not follow a specific text book. Different sources will be provided throughout the course in the form of scientific papers and book chapters.

For your own reference, the following material may be useful:

Deep Learning
Ian Goodfellow, Yoshua Bengio, Aaron Courville
MIT Press, 2016

Deep Learning with PyTorch
Vishnu Subramanian
Packt, 2018

Deep Learning
Ian Goodfellow, Yoshua Bengio, Aaron Courville
MIT Press, 2016

Deep Learning with PyTorch
Vishnu Subramanian
Packt, 2018

The course follows a traditional format and is held completely in the classroom.

Classes follow a hybrid format covering the theoretical as well as the more technical aspects of advanced machine learning. The hybrid nature of the classes lies in the frequent practical demos that are interleaved throughout the lectures. These tutorials are executed live and make up for about 40% of the course. The deep learning framework PyTorch will be used in these sessions. All students are required to actively engage in the tutorials, using their personal computers to work on the notebooks.

In presence, but it is not mandatory to attend the lectures.

Evaluation consists in the following steps:

1. A written midterm, acting as a self-evaluation test, which does not concur to the final grade.
2. A final written exam.
3. A project (not necessarily individual).
4. An optional oral exam.

The final grade is equal to 40% project plus 60% written exam.

The oral examination, which is optional, can bring up to 3 points (plus or minus) to the final score.

All these steps are aimed at evaluating the technical and theoretical skills, as well as the capability to work in a group, knowledge of the literature, the ability to formulate deep learning problems and to setup experiments.