By Ludovic Chappex

Poring over business news, scouring trends on social media, sifting through corporate activity reports, etc. What if this ritual, so familiar to the most thorough investors, was relegated to the history books? Seemingly unstoppable, artificial intelligence (AI) shows us a bit more every day its phenomenal ability to analyse and process a colossal flow of information, much faster and more efficiently than a human brain can. This miracle is made possible by Natural Language Processing (NLP), a branch of AI that uses machine learning to understand the structure and meaning of text.

Applied to the world of finance, this technology is used to analyse market sentiment, i.e., the general feeling of investors towards an asset or the market as a whole, virtually in real time. But can it predict changes in stock prices?

Results seem to suggest that it can. For example, the US Federal Reserve (Fed) – hardly known for its frivolity – has recently developed the Twitter Financial Sentiment Index (TFSI). In a research report entitled "More than Words: Twitter Chatter and Financial Market Sentiment", posted on its website in May, the Fed writes: "We build a new measure of credit and financial market sentiment using Natural Language Processing on Twitter data... We document that overnight Twitter financial sentiment helps predict next day stock market returns. Most notably, we show that the index contains information that helps forecast changes in the US monetary policy stance: a deterioration in Twitter financial sentiment the day ahead of an FOMC (ed. note: Federal Open Market Committee) statement release predicts the size of restrictive monetary policy shocks. Finally, we document that sentiment worsens in response to an unexpected tightening of monetary policy."

Other similar work positively evaluates market sentiment analysis based on press articles. Published in 2022 by two researchers from the Zurich University of Applied Sciences (ZHAW), the study "Using Financial News Sentiment for Stock Price Direction Prediction" tests natural language processing to extract market sentiment information from news articles and predict price direction of the S&P 500 stock market index. The authors conclude that "the results show that sentiment scores based on news content are particularly useful for stock price direction prediction".

Several tech companies have dived head first into the technology hype, offering turnkey AI-based market sentiment analysis tools to their clients (banks, hedge funds and pension funds). The California-based company MarketPsych (see CEO interview) and the Spanish-American firm RavenPack are among the pioneers and leaders in this field.

Put simply, these companies harness the millions of bits of news from the financial press, blogs and social media, and transform this data flow into actionable indices for investors. To achieve that, the first challenge is to build up a good database of sources with a reputation for being reliable. The next step is to define one or more themes using keywords, such as inflation, unemployment, COVID, recession, etc. 

These various themes are automatically detected by the NLP tools, which then generate statistics or a summary. The idea is to produce an assessment of the crowd’s feelings (positive, neutral or negative) towards a given theme or company. RavenPack boasts that it can deliver a snapshot of the market practically in real time. "The machine can collect data, analyse it and provide the client with an analysis in less than half a second," says company CEO Armando Gonzalez.

But how useful is market sentiment analysis? Can it replace traditional economic indicators? And more relevantly, what is its real predictive power? "These methods are reliable," says Norman Schürhoff, professor of finance at the University of Lausanne and the Swiss Finance Institute. "Ample evidence now confirms that news and social media provide valuable information. But market sentiment analysis should not be viewed as a substitute for traditional fundamental economic indicators. Rather, it complements and refines the predictive power of economic indicators."

This opinion echoes that of other academic experts that we contacted, such as Didier Sornette, professor emeritus at the Chair of Entrepreneurial Risks at ETH Zurich. "Market sentiment is like a photograph, a very detailed photograph, which can then be added to the toolbox to help predict prices and future risks."

While experts agree on the value of market sentiment analysis, they all highlight the crucial importance of the quality of the data processed by AI. RavenPack CEO Armando Gonzalez wholly agrees, "Storage capacity and computer speed have taken a giant leap forward, but the challenge remains isolating the right data. Good data is the basis for all models."

"The quality of the source is key, if you agglomerate information indiscriminately from the average of all influencers, you’ll lose money"

Norman Schürhoff, professor of finance at the University of Lausanne and the Swiss Finance Institute

In other words, simply taking in everything that is said on social media is an accident waiting to happen. Norman Schürhoff led a study published by the Swiss Finance Institute on 25 April, entitled "Finfluencers" (ed. note: portmanteau of financial influencers), which explains the phenomenon. This research estimates that 56% of influencers have "negative skill" or "antiskill" at predicting prices, generating -2.3% monthly abnormal returns, and only 28% are skilled, generating 2.6% monthly abnormal returns. "The advice by antiskilled finfluencers creates overly optimistic beliefs most times and persistent swings in followers’ beliefs," the authors write. "Consequently, finfluencers cause excessive trading and inefficient prices."

"The quality of the source is key," Norman Schürhoff says. "If you agglomerate information indiscriminately from the average of all influencers, you’ll lose money. But if you choose the best influencers as sources and filter this information using NLPs, the result is useful." He adds, "Taken as a whole, the predictive power of social media is more random than that of financial press articles, but our study shows that with the right methodology, you can obtain valuable information."

In addition to the careful data selection, the sheer quantity of data combined with maximum exploitation of NLP tools ultimately delivers the best performance. "Our research shows that the more data you aggregate, provided you choose the right data, and the more you use different NLPs, the higher the chance for outperformance," says Matthias Uhl, head of Analytics & Quant Modelling at UBS, who is also a lecturer at the University of Zurich and the Swiss Finance Institute.

Could fortune be at the end of the AI rainbow, provided you feed the machine good data? Amit Goyal, professor of finance at the University of Lausanne, doubts it. "With or without these technologies, a fundamental question remains: is it actually possible to predict prices? Several Nobel Prize winners in economics have argued that it’s simply not possible to predict changes in market prices. It’s a subject of ongoing debate among economists."

"There is no consensus on anything in economics," Didier Sornette says with a smile. "We’re divided into schools!" Relatively cautious about these new tools, the ETH Zurich professor continues, "AI and machine learning have become much more powerful, but the resulting improvement is more quantitative than qualitative. AI can provide ever more in-depth analyses, but it’s actually just deciphering the crowd’s stupidity (or intelligence)."

The challenge is not exactly the same between predicting prices in the short term and in the long term. "In the short term, behaviour is everything! The nearer in the future, the less sensitive price variation is to fundamentals," says Julien Leegenhoek, founder and CEO of Taranis, a Geneva-based company that uses AI to analyse alternative data (including market sentiment).

As for Matthias Uhl of UBS, he emphasises the importance of economic fundamentals when taking a long-term investment perspective. "Asset managers need indicators that are reliable over one, or even two, economic cycles, and I have not seen social media sentiment indicators work reliably over such a long period. That’s why it’s important for investors not to limit themselves to one speciality." On this note, Matthias Uhl adds the following anecdote, "I teach students who are sometimes experts in AI; some have mastered machine learning applications in finance, but these same students sometimes have no knowledge of how the economy works, which is equally important if you want to beat markets."

Paradoxically, the other problem with these technologies relates to their democratisation. A key notion in investing is having information that others don’t have, or don’t have yet. Even so, more and more companies are bringing their solutions to market, not to mention AI tools that anyone and everyone has access to, such as ChatGPT. The latest version of the ubiquitous conversational agent offers astounding results in stock-picking based on market sentiment. But its services are aimed at billions of potential investors, making it harder to stay a step ahead.

Matthias Uhl from UBS draws an interesting parallel: "Look at the P/E ratio (Price-Earnings Ratio), which most investors still look at today. The usefulness of this indicator has gradually gone down over the last decades, whereas in the 1970s and 1980s it proved to be a valuable indicator."

What’s in store for the future? "Market sentiment analysis will also cover video streams," says Armando Gonzalez, CEO of RavenPack. The idea is to identify specific behaviours." "The next step will be voice and video analysis of behavioural signals from CEOs and central bankers," agrees Norman Schürhoff of the University of Lausanne. "Machine learning methods will be developed to understand and explore individuals’ motivations. This type of research could help to make long-term forecasts."