From fiction to fact: the growing role of generative AI in business and finance

In one sentence

Summary

This paper addresses the emerging role of generative AI in financial decision-making, specifically focusing on its ability to extract actionable insights from unstructured corporate text. It positions generative AI not just as a tool for content creation but as a valid instrument for sentiment analysis and risk assessment, bridging the gap between natural language processing and quantitative finance. The authors aim to validate whether AI-derived sentiment can serve as a reliable predictor for firm behavior and market outcomes, contributing to the literature on AI applications in investment research. The methodology involves a case study using ChatGPT 3.5 and ChatGPT 4 to analyze sentiment towards California’s cap-and-trade program in the annual reports of 321 affected firms. The authors extract text mentioning the policy, input it into the models with a strict prompt template to generate negative sentiment scores on a 0-1 scale, and correlate these scores with financial data from Compustat. They employ regression models with firm fixed effects to test the relationship between AI-generated sentiment, R&D spending, leverage, and stock volatility, comparing the predictive power of ChatGPT 4 against ChatGPT 3.5 and traditional keyword mentions. The findings reveal that higher negative sentiment scores generated by ChatGPT 4 are significantly associated with increased R&D investment in cleaner technology and reduced leverage, indicating proactive risk management. Furthermore, these sentiment scores predict lower stock return volatility following the policy implementation, suggesting that investors reward firms that manage environmental regulatory risks effectively. The study highlights that ChatGPT 4 outperforms earlier versions and Bard in reasoning and consistency. However, it notes limitations such as prompt sensitivity and the need for regulatory frameworks to address ethical concerns and potential biases in AI-driven financial analysis.

Research problem

The paper investigates whether generative AI, specifically ChatGPT, can effectively analyze unstructured corporate text to extract sentiment scores that predict firm-level risk management behaviors and stock market performance. It addresses the gap in understanding how LLMs can be operationalized for financial sentiment analysis compared to traditional NLP methods.

Core contributions

• Validates ChatGPT 4 as a superior tool for financial sentiment analysis compared to ChatGPT 3.5 and Bard.
• Demonstrates that AI-generated sentiment scores from annual reports predict firm R&D and leverage adjustments.
• Shows that AI-captured negative sentiment towards environmental policy correlates with lower stock return volatility.
• Provides a replicable prompt template for extracting numerical sentiment scores from unstructured text.
• Highlights the sensitivity of LLM outputs to prompt variations, offering insights for robust AI application in finance.

Method

• Text extraction from annual reports using specific keyword searches.
• Prompt engineering with strict constraints to force numerical output from LLMs.
• Sentiment scoring using ChatGPT 3.5 and ChatGPT 4 on a 0-1 probability scale.
• Spearman Rank correlation to measure agreement between different model versions.
• Panel data regression with firm fixed effects to control for unobserved heterogeneity.
• Comparison of AI sentiment against traditional keyword-based dummy variables.
• Analysis of interaction terms between sentiment scores and policy implementation timing.

Results & findings

• ChatGPT 4 sentiment scores are highly correlated (0.970) with ChatGPT 3.5 scores, ensuring consistency.
• Firms mentioning cap-and-trade keywords increased R&D by 8.3% and reduced leverage by 55.3% post-policy.
• Higher negative sentiment scores predict significantly higher R&D inputs and further leverage reduction.
• AI-generated sentiment predicts lower stock return volatility, indicating better risk management perception.
• ChatGPT 4 outperforms Bard and ChatGPT 3.5 in reasoning, language understanding, and output stability.
• Prompt sensitivity is a key limitation; small changes in input can lead to significant output variations.

Data & experiments

• Sample of 321 listed companies in mining, utilities, construction, and manufacturing in California and neighboring states.
• Text data extracted from annual reports containing keywords 'cap and trade' or 'cap-and-trade' post-2011.
• Financial data from Compustat (2008-2017) including R&D, leverage, size, cash holdings, and ROA.
• Experiments comparing sentiment scores from ChatGPT 3.5, ChatGPT 4, and Bard on various business tasks.
• Regression analysis with firm fixed effects to test the impact of sentiment scores on financial metrics.
• Correlation analysis using Spearman Rank to assess consistency between different model outputs.

Limitations & caveats

• Study focuses on a specific policy (California cap-and-trade), limiting generalizability.
• Prompt sensitivity means results may vary with different input phrasing.
• ChatGPT outputs are not guaranteed to be factually accurate or unbiased.
• Sample is limited to specific industries and regions, potentially missing broader market effects.
• Does not account for other confounding factors that might influence R&D and leverage decisions.
• Relies on annual reports, which may not capture real-time sentiment changes.

Curator notes

Discovery status: round4_promoted. Source links: Large Language Models and Sentiment Analysis in Financial Markets: A Review, Datasets, and Case Study