3
$\begingroup$

I am just learning (more) about automatic differentiation (AD) and at this stage it kind of seems like black magic to me. The second paragraph of its Wikipedia article makes it sound too good to be true: it is extremely fast and is exact (no round off, no discretisation). I am left wondering why finite difference (FD) is so ubiquitous in scientific computing. Looking this up, I seem to only find tutorials on how to implement AD, the advantages of AD, and its applications in gradient-based optimisers. But what is an example of when not to use AD, and instead use FD? Surely there must many.

As just one example, in computational electromagnetics a FD approach is very standard; why can we not propagate Maxwell's equations with AD (FDTD: why not ADTD?)? It is clearly not because the developers aren't aware of it because the same people implement AD for inverse design purposes (why AD instead of FD for inverse design?). Naively, to me it seems like having an exact derivative should be more important when propagating Maxwell's equations than when taking the derivative of an objective function.

| cite | improve this question | |
$\endgroup$
  • $\begingroup$ I think you seem to be confused between FD and AD in terms of their use. $\endgroup$ – Chenna K 9 hours ago
6
$\begingroup$

Given code that computes a function $f(x)$, automatic differentiation tools produce a code that can compute $f(x)$ and its derivatives at the same time. Solving a differential equation is an entirely different problem and AD doesn't solve differential equations (although AD tools are sometimes useful in connection with PDE constrained optimization.)

AD tools are quite good at computing derivatives and should be used more often. However, there are circumstances where they simply can't be used. The most common reason that AD can't be used is that the code you have is a "black-box" for which source code is not available. Some other codes are so large and complicated that AD tools simply fail to handle them.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Ah, in hindsight that is actually pretty obvious, thanks. So with the exception of monstrously large functions, AD is basically always preferred to FD if we have the function f(x)? $\endgroup$ – tmph 19 hours ago
  • 1
    $\begingroup$ If you have code that implements $f(x)$, and that code is not too ugly, than AD tools are a good choice. However, if the source code isn't available or if the code is too complicated for your AD tools, then finite difference derivatives are generally the way to go. $\endgroup$ – Brian Borchers 5 hours ago

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.