To what extent is "advanced mathematics" needed/useful in A.I. research?

Question

I am currently studying mathematics. However, I don't think I want to become a professional mathematician in the future. I am thinking of applying my knowledge of mathematics to do research in artificial intelligence. However, I am not sure how many mathematics courses I should follow. (And which CS theory courses I should follow.)

From Quora, I learned that the subjects Linear Algebra, Statistics and Convex Optimization are most relevant for Machine Learning (see this question). Someone else mentioned that learning Linear Algebra, Probability/Statistics, Calculus, Basic Algorithms and Logic are needed to study artificial intelligence (see this question).

I can learn about all of these subjects during my first 1.5 years of the mathematics Bachelor at our university.

I was wondering, though, if there are some upper-undergraduate of even graduate-level mathematics subjects that are useful or even needed to study artificial intelligence. What about ODEs, PDEs, Topology, Measure Theory, Linear Analysis, Fourier Analysis and Analysis on Manifolds?

One book that suggests that some quite advanced mathematics is useful in the study of artificial intelligence is Pattern Theory: The Stochastic Analysis of Real-World signals by David Mumford and Agnes Desolneux (see this page). It includes chapters on Markov Chains, Piecewise Gaussian Models, Gibbs Fields, Manifolds, Lie Groups and Lie Algebras and their applications to pattern theory. To what extend is this book useful in A.I. research?

Answer 1

I do not want to sound condescending, but the math you are studying at the undergraduate and even graduate level courses is not advanced. It is the basics. The title of your question should be: Is "basic" math needed/useful in AI research? So, gobble up as much as you can, I have never met a computer scientist who complained about knowing too much math, although I met many who complained about not knowing enough of it. I remember helping a fellow graduate student in AI understand a page-rank-style algorithm. It was just some fairly easy linear algebra to me, but he suffered because he had no feeling for what eigenvalues and eigenvectors were about. Imagine the things AI people could do if they actually knew a lot of math!

I teach at a math department and I regularly get requests from my CS colleagues to recommend math majors for CS PhD's becase they prefer math students. You see, math is really, really hard to learn on your own, but most aspects of computer science are not. I know, I was a math major who got into a CS graduate school. Sure, I was "behind" on operating systems knowledge (despite having decent knowledge of Unix and VMS), but I was way, way ahead on "theory". It is not a symmetric situation.

Answer 2

Max, here is a (necessarily) partial list :

Basic linear algebra and probability are needed all over the place. I suppose you don't need references for that.

To my knowledge, Fourier analysis has been used in some learning-theory related investigation. Check out this paper, for instance.

The concept of manifold learning is getting popular, and you can start taking a look at the works of Mikhail belkin and Partha Niyogi. This line of work requires understanding of various concepts related to manifolds and riemannian geometry.

There is another aspect of machine learning, that has deeper roots into statistics, viz., Information geometry. This area ties in various concepts of Riemannian geometry, information theory, Fisher information, etc. A cousin of this sort of study can be found in Algebraic statistics - which is a nascent field with a lot of potential.

Sumio Watanabe, investigated a different frontier, viz., the existence of singularities in learning models and how to apply deep results of resolutions from algebraic geometry to address many questions. Watanabe's results draw upon heavily from Heisuke Hironaka's celebrated work that won him the Fields medal.

I suppose I am omitting many other areas that require relatively heavy math. But as Andrej pointed out, most of them probably do not lie at the frontiers of mathematics, but are relatively older and established domains.

At any rate, however, I suppose that the present state of AI that has entered into mainstream computing - such as in the recommendation systems in Amazon, or the machine learning libraries found in Apache Mahout, do not require any advanced math. I may be wrong.

Answer 3

Depends on your definition of advanced, and what sort of AI you want to study.

Many problems in AI are provably intractable-- optimal solutions to POMDPs are provably NP-Complete, optimal solutions to DEC-POMDPs are provably NEXP-Complete, etc. So, absent some unexpected breakthrough in complexity theory, the more one knows about approximation algorithms and their theoretical underpinnings, the better. (In addition to the measure theory, etc, needed to truly understand the Bayesian probability that underlies the POMDP model.)

Multi-agent artificial intelligence, in particular, intersects with game theory; so knowing game theory is helpful which in turn depends on topology, measure theory, etc. And likewise, many problems in game theory are intractable. Some are even intractable under approximation and even understanding when it is possible to usefully approximate takes a considerable amount of mathematics to work out.

(I note that the game theorists have been having a pretty good run in the Nobel Economics field, for the past few years, and that's heavily mathematical in nature. I predict in twenty odd years, today's algorithmic game theorists will be in about the same position.)

Answer 4

The maths involved in AI are not advanced, and are taught at the undergrad level. AI training and inferencing algorithms are in the domain of advanced Computer Science.

It is a bit of a word game. Some history should also be included when researching AI.

For example, in the current nomenclature, Deep Learning seems to be a trending keyword in AI.

Deep Learning is what used to be referred to as Artificial Neural Networks (ANNs) such as Hinton's backpropagating perceptron network model (BACKPROP), and the like.

The maths involved with a BACKPROP ANN (for example) are essentially derivative calculus for training, and matrix algebra for inferencing.

The new aspect of Deep Learning is the physical separation of training and inferencing algorithms. CPUs are still used for training, but now GPUs are used for inferencing.

For example, ANN matrices are trained (weighted) by backpropagating errors using corrective derivative calculus. This is best suited to CPUs, and only has to be performed once per ANN deployment.

The ANN is then deployed in a highly parallelized GPU architecture. The forward inferencing math involves intensive matrix algebra, which GPUs are designed for.

This boosts performance of a deployed ANN by several orders of magnitude compared to previous CPU-based deployments, and can be more efficiently scaled across any number of dedicated GPUs.

Companies such as Nvidia and AMD are now marketing very high end GPU chipsets as Deep Learning Machines. The term GPU has always been a bit of a misnomer, since they are really general purpose Parallel Processors. For example, GPUs are also sometimes referred to as Bitminers in blockchain applications.

So what was old is now new. The maths involved have not changed, just the terminology of the Computer Science (mostly due to marketing pressures).

The term AI has always been considered a bit of a dark horse. Deep Learning is now the politically correct, market friendly term.

Answer 5

AI is an amazingly broad field with a wide range of possible routes. Some are extremely mathematical, some barely touch mathematics. Others have already given good answers for the more mathematical approaches. Of the subjects you pointed out-

"Linear Algebra, Probability/Statistics, Calculus, Basic Algorithms and Logic"

-you do basically need or will benefit from all of them. Many approaches are at least partly directly based on probability and statistics - heuristics, neural networks, genetic algorithms, fuzzy logic. Calculus is equally useful - in AI or in general computing science you will find it almost everywhere. Linear algebra is something you definitely need as well.

The two most essential subjects from a CS/AI perspective are algorithms and logic, algorithms are the real heart of computing science, and logic is the underlying 'language' of algorithms.. The key to learning algorithms though is learning how to program, proficiency and practice at basic programming is one of the most important foundations of almost all computer science or AI subjects. Programming is also a skill that universities are not always particularly good at teaching. Logic is also really essential to most branches of AI; Boolean logic, predicate calculus, symbolic logic, underlying theories of permutation, the hierarchy of design, recursion, finite state machines, Turing Machines, CPU design, etc.. Here we are really stepping away from mathematics into computing science proper..

Extending to my own field of 'Strong AI' mathematics plays an underlying but absolutely essential role. A very good understanding of basic maths is probably more important than higher maths, but really anything you pick up can be useful. The real problem in a nascent field like Strong AI is that everything is up in the air and so the field is in total flux.
Subjects that are potentially useful include - neural networks, genetic algorithms, neurology, genetics, psychology, cybernetics and robotics, 3D graphics theory, image processing theory, computer games design, philosophy, art theory, digital electronics, linguistics theory.. In a field like this reading is one of the most important ways to learn. A couple of books that were starting points for me were - The Emperors New Mind by Roger Penrose, Eye and Brain by RL Gregory, but really insights can come from almost anywhere