Mathematical Principles of Deep Learning

Course materials

A preliminary course syllabus can be found here. Content will be added throughout fall 2022, often in the form of links to resources upon which the presentation is based. Any materials posted here are publically available for non-commercial use. In particular, they are not to be shared on for profit sites such as 'Course Hero'.

1. Introduction

2. Approximation by neural networks

3. Optimization for neural networks

4. Practical aspects

5. Generalization

6. Additional topics in deep learning

There are many events at TAMU in the context of deep learning, general machine learning, and data science, which may be of interest to you. Some are listed here.

Homework

Homework may be modified to fix errors, unclear formulations, or overly difficult assignments. If you find any of the above, please email me.

1. Approximation by neural networks

2. ReLU networks

3. Optimization

4. Implementation

5. Advanced optimization

Additional (free and accessible) resources:

• Lecture notes by Matus Telgarsky

• Lecture notes by Tengyu Ma

• Handwritten notes for a Course on the Mathematics of Deep Learning by Joan Bruna

• A video recording of a presentation by Tyrus Berry, which touches on a lot of topics in data science and machine learning in an intuitive way

• A high-bias, low-variance introduction to Machine Learning with associated Python notebooks (which can be run online). Note: The Tensorflow notebooks use Tensorflow 1, not Tensorflow 2. There are quite a few changes between the two generations.

• Google Colab is a free online Python environment in which you can easily run code with the most common libraries for machine learning. It supports GPU processing, but will limit your processing power if your use of computing resources becomes very high.

• There are several recent overview articles over the mathematical theory of deep learning:

  • Higham and Higham (2018)

  • E, Ma, Wojtowytsch and Wu (2020)

  • Berner, Grohs, Kutyniok and Petersen (2021)

• A textbook on deep learning (with a focus for non-mathematicians)

• An introduction by Roberts, Yaida and Hanin (2021)

• A textbook on machine learning theory in general for additional background.

• Lecture notes on high-dimensional probability by Ramon van Handel for additional background.

• A text book on high-dimensional probability by Roman Vershynin for additional background.

I am grateful for further sources or amterials. Please email me with suggestions for links to add to the list.