Summary
In this case study, Mosaic fine-tuned language models in quantifying commodity sentiment with AI to detect drivers of pricing. Our team used sentiment model outputs to extract insights regarding price trends for a specific commodity and predict future price movements.
Take Our Content to Go
An Introduction to Sentiment Analysis
Sentiment Analysis is a core component of Natural Language Processing (NLP). Such models attempt to quantify the emotion expressed in a piece of text along a spectrum of negative to neutral to positive. The output of many sentiment models is typically a series of three probabilities: the probability the text is positive, the likelihood the text is negative, and the probability the text is neutral. The following examples use the Huggingface distilbert-base-multilingual-cased-sentiments-student Sentiment model:
Example 1:
“Today was so much fun. I cannot wait until tomorrow.”
Model output: positive: 0.718, neutral: 0.179, negative: 0.103
Example 2:
“I got a paper cut, and it really hurt.”
Model output: negative: 0.791, positive: 0.132, neutral: 0.077
Example 3:
“I just finished eating lunch.”
Model output: positive: 0.389, neutral: 0.339, negative: 0.273.
One of the more challenging tasks of sentiment analysis arises when text contains niche or industry-specific keywords unfamiliar to the model. A sentiment model created from generalizable language may produce subpar sentiment probability scores when applied to text with industry-specific words. This issue can be rectified by taking a generalized large language model (LLM) and finetuning the model with a corpus of text related to the industry.
The result of this industry-specific model is a model tuned to the industry keywords and appropriate emotions associated with these keywords. One of the more notable applications of such approaches resulted in “FinBERT,” a large language model based on BERT but fine-tuned to specialized language used in a financial context.
The Challenge of Price Prediction
Predicting stock and/or commodity prices based solely on text sentiment is not reasonably expected due to the strong influence of various macroeconomic factors on the stock market. However, many industries and companies within the public eye can attest to instances where the price of their stock/commodity trended outside reasonable expectations from macroeconomic factors alone.
Along with the macro and company-specific economic factors, there appear to be psychological and sociological factors, such as how the public views an industry and/or commodity, which can affect price fluctuations in the short term. A 2021 study looked at reported news specific to 87 different companies over a span of seven years to conclude that “there exists a weak but statistically significant association between strong media sentiment and abnormal market return as well as volatility.” In other words, when it comes to price fluctuations, sentiment matters.
For this reason, Mosaic Data Science uses large language models to capture the short-term influence of sentiment and analyzes how it may help explain and/or predict specific commodity price movements.
Commodity Pricing With AI Solution
In the following case study, Mosaic applied sentiment modeling to help a large company capture as much of the conversation as possible surrounding specific commodities and use sentiment to explain short-term price movements. Commodity prices are influenced by many factors, encompassing macroeconomic elements like interest rates, inflation, country GDP, and commodity-specific factors such as supply and demand dynamics and company financial reports.
Mosaic’s customer operates as a global manufacturer of various commodities. The customer aimed to explore multiple time series methodologies for predicting future commodity prices, enhancing its financial decision-making processes. Engaged in an industry with high visibility, the customer believed public sentiment could help predict fluctuations in commodity prices. They sought a more systematic method to capture and understand these fluctuations effectively.
Mosaic Data Science was the ideal partner for this effort. Given the team’s expertise in Natural Language Processing, sentiment analysis, and the deployment of advanced machine learning models, Mosaic brought relevant expertise to support our customers’ efforts to explain commodity price fluctuations.
Creating Text Corpus for Commodity Pricing With AI
Before starting model development, Mosaic collaborated with the customer to pinpoint sources where market sentiment is most often expressed to capture as much of the conversation as possible. Consequently, Mosaic gathered content from industry analyst reports, news outlets, social media, and internal company emails.
Mosaic generated a set of commonly occurring topics from the extracted text and categorized each topic as relevant or irrelevant with input from the customer. Subsequently, each piece of text underwent processing through a BERT-based language model, receiving labels based on the topics present in the text. These topic labels were then used to construct a text corpus pertinent to the company and the specific commodity of interest. Managing over 1.5 million pieces of text, applying deep learning for topic assignment, and filtering irrelevant text before sentiment modeling ensured that sentiment indices remained unbiased, avoiding emotionally charged yet irrelevant data’s influence on the industry or focused commodity.
Image 1 shows how the strength of topic sentiment over time correlates with fluctuations in overall text sentiment. For instance, sentiment regarding Topic 4 is often positive (negative) while the general sentiment index is rising (falling). Using this correlation analysis, Mosaic could better assess how different topics impact public sentiment.
Automating Sentiment Labels
Defining text sentiment becomes relatively straightforward when employing a generic sentiment model, mainly when the text includes explicit emotion-laden words like “love,” “hate,” “good,” and “bad.” However, this project’s complexity arises from discerning how to quantify “sentiment” in terms of predicting changes in a specific commodity price.
For instance, consider a scenario where a company is interested in predicting the future price of apples. A text containing the statement “I love apples!” would receive a high positive sentiment probability, suggesting an anticipated increase in apple demand and, consequently, a rise in apple prices. Now, let’s examine another case. A text containing the sentence “I love oranges!” might also be labeled with a high positive sentiment probability.
However, when extending this sentiment to the future price of apples, one might infer a prediction of decreased apple demand as consumers put more of their handheld fruit budget toward oranges and, consequently, a decline in future apple prices. These two examples emphasize two different ways to quantify text sentiment when concerned with the price of apples:
- Emotional Sentiment: Refers to our conventional understanding of sentiment; measured on a scale ranging from extremely negative to extremely positive.
- Market Direction Sentiment: Pertains to how the emotional sentiment, when applied to the text’s context, influences the price of a specific commodity, measured on a scale ranging from decreasing the price to increasing the price.
After systematically refining the corpus to include only the most relevant text, Mosaic distributed a random sample of the corpus to internal industry experts. Their task was to label the text for both emotional sentiment and market direction sentiment. Leveraging these labeled text samples, Mosaic trained two distinct sentiment models, refining them from a generic sentiment model. The training process began with a powerful open-source, transformer-based large language model (LLM) pre-trained on text discussions related to stock market data and consequently familiar with keywords crucial for assigning sentiment to text related to generic business and market behaviors.
Our internal labels and training procedures enhanced the generic large language model (LLM), tailoring it to incorporate topic and market behavior specifics related to the customer. The fine-tuning processes resulted in one model quantifying emotional sentiment and another model quantifying market direction sentiment concerning the commodity of interest. These models offered a more systematic approach to processing industry-specific text, capable of being deployed as part of automated pipelines for continued text assessment.
Aggregating Sentiment Labels
For every piece of text, each fine-tuned sentiment model produced a series of three labels and their respective probabilities, indicating the likelihood the text belonged to each label. For the emotional sentiment model, the three output labels were “positive,” “neutral,” and “negative.” In contrast, for the market direction sentiment model, the three output labels were “increase,” “no movement,” and “decrease.”
Mosaic used these probability outputs to create two sentiment indices:
- Emotional Sentiment Index: Negative Probability – Positive Probability
- Market Direction Sentiment Index: Down Probability – Up Probability
Both indices were scaled between -1 and 1. Mosaic computed the respective sentiment indices by text source and date so that weekly commodity prices could be modeled as a function of source emotional and market direction sentiment. Image 2 visualizes a subset of these indices. Over the displayed two-year period, the commodity price often (but not always) follows the sentiment of different sources. Mosaic used these historical sentiment patterns as features to predict future price movements.
Defining Price Prediction Needs
Using the source-specific indices, Mosaic was curious about how the magnitude and timing of sentiment influenced price fluctuations. For example, if public sentiment was positive yesterday, would this positively affect future commodity prices, and if so, how long would it take before the price experienced a significant movement?
To meet our customer’s business goals, Mosaic needed to understand the relationship between text sentiment shifts several weeks before shifts in price. Image 3 shows one of the correlation analyses Mosaic conducted to understand when price and sentiment trended together and, if so, for how long. As seen in Image 3, when the lagged index from this particular source moved in the same direction as the price, the image shows seasons of high correlation. Learning from these source-specific patterns, Mosaic was able to better understand how feelings are not necessarily siloed to the weeks in which they first appear but often build over time and have lasting effects.
These analyses guided Mosaic in enhancing features of the sentiment index, allowing it to predict commodity prices up to eight weeks in advance.
Quantifying Commodity Sentiment with AI Results
The model’s outputs and insights derived from the modeling process empowered Mosaic to provide the customer with information that could be utilized to identify trends related to text topics, public sentiment, and the connections between commodity prices and text, considering the text source and the intensity of sentiment.
Is Quantifying Commodity Sentiment with AI Possible?
This case study underlines the intricate relationship between public sentiment and commodity prices, where Mosaic’s methodologies not only identified shifts in public sentiment but also correlated these changes to commodity price movements. Mosaic employed a deep learning topic model to construct a relevant text corpus for their customer, fine-tuned two large language sentiment models tailored to the customer’s industry data, and applied these models to analyze over five years’ worth of data, tracking shifts in public sentiment over time. By integrating the outcomes of our fine-tuned sentiment models with commodity price information, we examined the correlation between sentiment and price changes, demonstrating the powerful capabilities of these models in extracting nuanced insights from vast amounts of data.
The result of Mosaic’s efforts included tuned models adept at handling industry-specific text and an analysis of historical sentiment and price fluctuations. This analysis can inform future business decisions related to commodity prices. Thanks to Mosaic’s work, the customer directly uses these predictions to optimize their manufacturing plans, drive more margin and volume by pricing ahead of market shifts, and become the first-mover in their market. The success of this project not only benefits Mosaic’s client but also sets a precedent in the financial sector, showcasing the potential of advanced data science and NLP techniques in transforming market analysis and decision-making processes.
