A Step-by-step Introduction to the Implementation of Automatic Differentiation
CoRR(2024)
摘要
Automatic differentiation is a key component in deep learning. This topic is
well studied and excellent surveys such as Baydin et al. (2018) have been
available to clearly describe the basic concepts. Further, sophisticated
implementations of automatic differentiation are now an important part of
popular deep learning frameworks. However, it is difficult, if not impossible,
to directly teach students the implementation of existing systems due to the
complexity. On the other hand, if the teaching stops at the basic concept,
students fail to sense the realization of an implementation. For example, we
often mention the computational graph in teaching automatic differentiation,
but students wonder how to implement and use it. In this document, we partially
fill the gap by giving a step by step introduction of implementing a simple
automatic differentiation system. We streamline the mathematical concepts and
the implementation. Further, we give the motivation behind each implementation
detail, so the whole setting becomes very natural.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要