15
$\begingroup$

I am currently studying mathematics. However, I don't think I want to become a professional mathematician in the future. I am thinking of applying my knowledge of mathematics to do research in artificial intelligence. However, I am not sure how many mathematics courses I should follow. (And which CS theory courses I should follow.)

From Quora, I learned that the subjects Linear Algebra, Statistics and Convex Optimization are most relevant for Machine Learning (see this question). Someone else mentioned that learning Linear Algebra, Probability/Statistics, Calculus, Basic Algorithms and Logic are needed to study artificial intelligence (see this question).

I can learn about all of these subjects during my first 1.5 years of the mathematics Bachelor at our university.

I was wondering, though, if there are some upper-undergraduate of even graduate-level mathematics subjects that are useful or even needed to study artificial intelligence. What about ODEs, PDEs, Topology, Measure Theory, Linear Analysis, Fourier Analysis and Analysis on Manifolds?

One book that suggests that some quite advanced mathematics is useful in the study of artificial intelligence is Pattern Theory: The Stochastic Analysis of Real-World signals by David Mumford and Agnes Desolneux (see this page). It includes chapters on Markov Chains, Piecewise Gaussian Models, Gibbs Fields, Manifolds, Lie Groups and Lie Algebras and their applications to pattern theory. To what extend is this book useful in A.I. research?

$\endgroup$
  • 21
    $\begingroup$ in my 2+ years on this site i've seen more than half a dozen questions of the type "what math do i need for...". Que answers that read like the contents of the Princeton Companion to Mathematics. 1) AI is a huge field, the math used in its subfields comes in all flavors; 2) Take your basic math courses, choose more advanced courses based on interest; 3) Do research in AI, find what you like, find what math is used there; 4) We cannot know apriori what math will be useful for this or that problem. $\endgroup$ – Sasho Nikolov Oct 19 '12 at 23:48
55
$\begingroup$

I do not want to sound condescending, but the math you are studying at the undergraduate and even graduate level courses is not advanced. It is the basics. The title of your question should be: Is "basic" math needed/useful in AI research? So, gobble up as much as you can, I have never met a computer scientist who complained about knowing too much math, although I met many who complained about not knowing enough of it. I remember helping a fellow graduate student in AI understand a page-rank-style algorithm. It was just some fairly easy linear algebra to me, but he suffered because he had no feeling for what eigenvalues and eigenvectors were about. Imagine the things AI people could do if they actually knew a lot of math!

I teach at a math department and I regularly get requests from my CS colleagues to recommend math majors for CS PhD's becase they prefer math students. You see, math is really, really hard to learn on your own, but most aspects of computer science are not. I know, I was a math major who got into a CS graduate school. Sure, I was "behind" on operating systems knowledge (despite having decent knowledge of Unix and VMS), but I was way, way ahead on "theory". It is not a symmetric situation.

$\endgroup$
  • 6
    $\begingroup$ measure theory and probability theory are the basis for all of probabilistic reasoning. topology has become very important for topological data analysis. Fourier analysis is important for learning theory (it's used to understand the sensitivity of functions and how difficult it is to learn them), and manifold learning requires a deep understanding of manifold geometry. $\endgroup$ – Suresh Venkat Oct 19 '12 at 22:17
  • 2
    $\begingroup$ @MaxMuller: And to continue the list, group theory and algebraics (like Lie algebras) are used extensively in pattern recognition in the decomposition theory of images where topology is heavily required (and there is a deep connection between Lie algebras and manifolds you need to learn along the way). Books such as Monique Pavel's "Fundamentals of Pattern Recognition" will even introduce you to category theory and it's application, which also is extremely important in AI for it's use in the foundations of formal languages and proof theory (which can be a theory of reasoning)... $\endgroup$ – ex0du5 Oct 19 '12 at 23:16
  • 1
    $\begingroup$ Past intro level grad courses, mathematicians learn all their math on their own (or in reading groups and seminars)..it's not all that hard if you have some foundations...ok, it can be hard, but it's not impossible. $\endgroup$ – Sasho Nikolov Oct 19 '12 at 23:52
  • 1
    $\begingroup$ Max, I teach aikido as well. I do not recall any students of aikido asking "why do I have to learn the basics (how to fall, how to move from the line of attack)?" Sometimes you need to trust a bit that your teachers know what they are doing. I will however be the first to admit that we teach a lot of crap, especially in high schools and primary schools where math is taught as if the purpose was to stifle curiosity in students. But in your case, the subjects you listed, they are not crap. Trust me. $\endgroup$ – Andrej Bauer Oct 21 '12 at 3:15
  • 13
    $\begingroup$ And just one more comment. If you learn only math that had already been proven useful in some area of CS, you will never have a chance to apply new math. You will always be behind. Science is an art, not a 9-to-5 job. If you ask me "should I learn physics, I want to get into AI" I will say "absolutely so!" And if you ask "should I learn sociology, I want to get into AI" my answer will still be the same. $\endgroup$ – Andrej Bauer Oct 21 '12 at 3:17
6
$\begingroup$

Max, here is a (necessarily) partial list :

Basic linear algebra and probability are needed all over the place. I suppose you don't need references for that.

To my knowledge, Fourier analysis has been used in some learning-theory related investigation. Check out this paper, for instance.

The concept of manifold learning is getting popular, and you can start taking a look at the works of Mikhail belkin and Partha Niyogi. This line of work requires understanding of various concepts related to manifolds and riemannian geometry.

There is another aspect of machine learning, that has deeper roots into statistics, viz., Information geometry. This area ties in various concepts of Riemannian geometry, information theory, Fisher information, etc. A cousin of this sort of study can be found in Algebraic statistics - which is a nascent field with a lot of potential.

Sumio Watanabe, investigated a different frontier, viz., the existence of singularities in learning models and how to apply deep results of resolutions from algebraic geometry to address many questions. Watanabe's results draw upon heavily from Heisuke Hironaka's celebrated work that won him the Fields medal.

I suppose I am omitting many other areas that require relatively heavy math. But as Andrej pointed out, most of them probably do not lie at the frontiers of mathematics, but are relatively older and established domains.

At any rate, however, I suppose that the present state of AI that has entered into mainstream computing - such as in the recommendation systems in Amazon, or the machine learning libraries found in Apache Mahout, do not require any advanced math. I may be wrong.

$\endgroup$
2
$\begingroup$

Depends on your definition of advanced, and what sort of AI you want to study.

Many problems in AI are provably intractable-- optimal solutions to POMDPs are provably NP-Complete, optimal solutions to DEC-POMDPs are provably NEXP-Complete, etc. So, absent some unexpected breakthrough in complexity theory, the more one knows about approximation algorithms and their theoretical underpinnings, the better. (In addition to the measure theory, etc, needed to truly understand the Bayesian probability that underlies the POMDP model.)

Multi-agent artificial intelligence, in particular, intersects with game theory; so knowing game theory is helpful which in turn depends on topology, measure theory, etc. And likewise, many problems in game theory are intractable. Some are even intractable under approximation and even understanding when it is possible to usefully approximate takes a considerable amount of mathematics to work out.

(I note that the game theorists have been having a pretty good run in the Nobel Economics field, for the past few years, and that's heavily mathematical in nature. I predict in twenty odd years, today's algorithmic game theorists will be in about the same position.)

$\endgroup$
1
$\begingroup$

The maths involved in AI are not advanced, and are taught at the undergrad level. AI training and inferencing algorithms are in the domain of advanced Computer Science.

It is a bit of a word game. Some history should also be included when researching AI.

For example, in the current nomenclature, Deep Learning seems to be a trending keyword in AI.

Deep Learning is what used to be referred to as Artificial Neural Networks (ANNs) such as Hinton's backpropagating perceptron network model (BACKPROP), and the like.

The maths involved with a BACKPROP ANN (for example) are essentially derivative calculus for training, and matrix algebra for inferencing.

The new aspect of Deep Learning is the physical separation of training and inferencing algorithms. CPUs are still used for training, but now GPUs are used for inferencing.

For example, ANN matrices are trained (weighted) by backpropagating errors using corrective derivative calculus. This is best suited to CPUs, and only has to be performed once per ANN deployment.

The ANN is then deployed in a highly parallelized GPU architecture. The forward inferencing math involves intensive matrix algebra, which GPUs are designed for.

This boosts performance of a deployed ANN by several orders of magnitude compared to previous CPU-based deployments, and can be more efficiently scaled across any number of dedicated GPUs.

Companies such as Nvidia and AMD are now marketing very high end GPU chipsets as Deep Learning Machines. The term GPU has always been a bit of a misnomer, since they are really general purpose Parallel Processors. For example, GPUs are also sometimes referred to as Bitminers in blockchain applications.

So what was old is now new. The maths involved have not changed, just the terminology of the Computer Science (mostly due to marketing pressures).

The term AI has always been considered a bit of a dark horse. Deep Learning is now the politically correct, market friendly term.

$\endgroup$
  • 2
    $\begingroup$ A previous answer has already given counterexamples to your claim in the first sentence. (There are many others as well.) Did you read the prior answers before posting? You might want to edit this answer to narrow your claims. $\endgroup$ – D.W. May 24 '17 at 21:03
  • 2
    $\begingroup$ Your claim that "CPUs are still used for training [deep networks], but now GPUs are used for inferencing" is rather misleading (if not incorrect). Everyone trains modern neural networks are on GPUs. Most people deploy them on GPUs as well, but some deployment environments (e.g. some cellphones) still use CPUs. $\endgroup$ – Mike Izbicki Jul 26 '17 at 21:50
  • $\begingroup$ I agree with Mike. "GPUs are used for training and CPUs for inferencing" is less incorrect than saying that "CPUs are used for training and GPUs for inferencing" $\endgroup$ – ASDF Aug 20 '17 at 14:41
  • $\begingroup$ @MikeIzbicki Pipeline architectures such as CUDA, OpenCL, etc are required for training in Deep Learning, which heavily relies on CPU cores for error correction. Inference pipelines only require CPU cores to feed and harvest the GPU cores. Power and thermal efficiency is the goal, which is why the balance between the core types shifts between training and inferencing. Which is what I already said. $\endgroup$ – Dominic Cerisano Nov 23 '17 at 6:03
-1
$\begingroup$

AI is an amazingly broad field with a wide range of possible routes. Some are extremely mathematical, some barely touch mathematics. Others have already given good answers for the more mathematical approaches. Of the subjects you pointed out-

"Linear Algebra, Probability/Statistics, Calculus, Basic Algorithms and Logic"

-you do basically need or will benefit from all of them. Many approaches are at least partly directly based on probability and statistics - heuristics, neural networks, genetic algorithms, fuzzy logic. Calculus is equally useful - in AI or in general computing science you will find it almost everywhere. Linear algebra is something you definitely need as well.

The two most essential subjects from a CS/AI perspective are algorithms and logic, algorithms are the real heart of computing science, and logic is the underlying 'language' of algorithms.. The key to learning algorithms though is learning how to program, proficiency and practice at basic programming is one of the most important foundations of almost all computer science or AI subjects. Programming is also a skill that universities are not always particularly good at teaching. Logic is also really essential to most branches of AI; Boolean logic, predicate calculus, symbolic logic, underlying theories of permutation, the hierarchy of design, recursion, finite state machines, Turing Machines, CPU design, etc.. Here we are really stepping away from mathematics into computing science proper..

Extending to my own field of 'Strong AI' mathematics plays an underlying but absolutely essential role. A very good understanding of basic maths is probably more important than higher maths, but really anything you pick up can be useful. The real problem in a nascent field like Strong AI is that everything is up in the air and so the field is in total flux.
Subjects that are potentially useful include - neural networks, genetic algorithms, neurology, genetics, psychology, cybernetics and robotics, 3D graphics theory, image processing theory, computer games design, philosophy, art theory, digital electronics, linguistics theory.. In a field like this reading is one of the most important ways to learn. A couple of books that were starting points for me were - The Emperors New Mind by Roger Penrose, Eye and Brain by RL Gregory, but really insights can come from almost anywhere

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.