Ask a Caltech Expert: Yaser Abu-Mostafa on AI and Finance

Can AI Predict Economic Downturns?

There has been a symbiotic relationship between financial institutions and the artificial intelligence community since the 1980s. That's when the field of neural networks started in earnest. Although there was lots of activity, we can see in hindsight that neural networks did not reach their full potential owing to the lack of computational resources and lack of abundant data.

Finance probably has the best set of data of any field, and that's been true for decades. So, when financial institutions came to researchers asking how AI could be applied to markets, it occurred to me that this was an opportunity not only for finance but also for AI and machine learning proper. Of course, on their end, they wanted to accurately predict what the market would do so that people could make money. On my end, I could improve machine learning methods with access to these data. I was able to run data on credit-card approval, forecasting foreign exchange rates, and all kinds of financial instruments.

While other applications of neural networks struggled to make progress, there was quite a bit of activity in finance. I would say it was the most impressive application of neural networks for 5 to 10 years. But a lot of it was proprietary. It was not shared widely, so the level of publication was not proportionate to the level of activity.

Today, with more sophisticated machine learning models, greater computational power, and huge, well-labeled sets of data, people are revisiting all sorts of old applications of neural networks in order to achieve even more. Finance just had a head start.

For most investment funds, there is a portion of trading activity that is automated using AI. It's an integral part of investment, trading, and other banking activities at almost every major bank. For example, trading algorithms compare the amount of money they expect to make versus the risk they are taking by investing a certain amount of money in the stock market, and investors use these systems to assess and predict risk in real time.

What these systems are not very good at predicting are the anomalies, like the 2008 crash. There are not enough existing, historical data points of crashes to train the model to recognize the patterns leading up to those events and predict when another will happen. For smaller perturbations in the market, like recessions, we are just recently getting enough data to predict those. Recessions happen with some regularity. I worked with Paraconic Technologies Ltd to access rich historical data about the stock market and was able to create a model that predicted recessions. My colleagues and I used a type of model that is tailored for situations where there are not too many data points—in this case, recessions—to learn from.

Theoretically, large-scale use of AI in finance would make markets completely efficient—meaning no one could "beat" the market—but that is an idealized notion. For that to happen, everyone interacting with the market would need to act fully rationally and have full information about what is happening in the market. But of course, people are not rational or all knowing. This is where the idea of fully automated trading comes from. There is promise and progress toward that, but we are not there yet. I would not take financial advice for my retirement plan solely from AI, yet.

— Yaser Abu-Mostafa, professor of electrical engineering and computer science

You can submit your own questions to the Caltech Science Exchange.