12-04, 18:00–18:30 (UTC), General Track
This talk is an introduction to automatic differentiation with a focus on the Python and Julia ecosystems. We will first explain what autodiff is and how it works, then describe its various implementations in both languages. Our goal is to give everyone a good understanding of how computer code can be differentiated, while also discussing the trade-offs this differentiability entails.
It's 2024, are you still computing gradients by hand? Then let's free up some of your time for more rewarding hobbies! Thanks to automatic differentiation, you can build awesome machine learning models and let the computer worry about derivatives. Well... mostly.
Not every piece of code is amenable to autodiff. Writing differentiable programs is a non-trivial task, which can only be mastered knowing what the computer does under the hood. It is also essential to understand the features and limitations of each machine learning stack.
The first part of the talk will explain how computer code is transformed to compute derivatives. We will clarify the sometimes confusing terminology of this field (numeric / symbolic / algorithmic differentiation) and push back against common misconceptions (gradients obtained by autodiff are usually exact up to floating point errors!). We will also discuss the difference between forward and reverse mode and the extension to higher-order derivatives.
The second part of the talk will provide a comparison between two high-level programming languages with excellent autodiff ecosystems: Python and Julia. We will cover Python's trinity of autodiff frameworks (TensorFlow, PyTorch, JAX) and show that the situation in Julia is very different: the code is not constrained by the choice of autodiff. In particular, there is no need to consider only a certain type of tensors, or a certain family of operations. We will comment on this crucial distinction, analyzing its upsides and downsides. Finally, we will give some recommendations on how to pick the right language and autodiff library for your own projects.
This presentation is based partly on research work with Adrian Hill.
No previous knowledge expected
Guillaume Dalle (https://gdalle.github.io/) is a postdoctoral researcher at EPFL who specializes in machine learning and combinatorial optimization. He is a prolific contributor to the Julia package ecosystem, especially for automatic differentiation and graph theory.