Quantifying Material Risks from Textual Disclosures in Financial Statements using Large Language Model Agents

In one sentence

Summary

The paper addresses the challenge of extracting and evaluating qualitative physical risk disclosures (e.g., natural disasters) from financial statements, where traditional keyword-based NLP fails to capture indirect or context-dependent language. It positions LLM agents as a superior tool for context-aware materiality assessment, moving beyond simple sentiment analysis to evaluate the financial significance of risks relative to a specific firm's profile. The research highlights the need for supervisors and investors to distinguish between material and non-material risks to improve decision-making and financial stability monitoring. The methodology employs a three-stage pipeline: first, using semantic embeddings (OpenAI text-embedding-3-small) to extract relevant sentences from SEC 10-K and 8-K filings; second, applying sentiment analysis (ProntoNLP) to classify risk tone; and third, deploying an LLM agent to assess materiality. The agent is prompted with the disclosure text alongside granular company data, including financials (assets, revenue, EBITDA), business description, industry, and location. The study uses case studies from Box 3 and Table 1 to illustrate how the agent weighs factors like financial health and operational vulnerability to determine if a risk is material. Findings show that the LLM agent effectively differentiates materiality based on context, such as identifying an $8.7 million write-off as material for a weak firm but a $69.2 million loss as non-material for a larger entity. It also recognizes long-term trends (e.g., increasing wildfires) as material for vulnerable sectors like winemaking. The paper concludes that LLMs augment human expertise by providing tailored initial evaluations, though final judgment remains with experts. Limitations include the subjective nature of materiality and the reliance on accurate external data integration.

Research problem

Traditional NLP methods struggle to extract and evaluate qualitative physical risk disclosures in financial statements because they rely on keyword matching and lack contextual understanding. This leads to missed risks described indirectly or an inability to assess whether a disclosed risk is financially material to a specific company. The paper addresses the need for a scalable, context-aware method to assess the materiality of physical climate risks for investors and regulators.

Core contributions

• Proposes a three-stage LLM-based pipeline for physical risk analysis: extraction, sentiment, and materiality assessment.
• Demonstrates the use of semantic embeddings to capture indirect risk language beyond keyword lists.
• Introduces an LLM agent framework that integrates textual disclosures with firm-specific financial and operational data.
• Provides empirical case studies showing how LLM agents differentiate materiality based on company context.
• Highlights the limitations of traditional keyword and sentiment-only approaches in risk assessment.

Method

• Semantic extraction using OpenAI text-embedding-3-small model.
• Dimensionality reduction using t-SNE for visualization.
• Sentiment classification using ProntoNLP service.
• LLM agent prompting with structured company data (financials, location, industry).
• Materiality judgment based on financial health, operational reliance, and geographic exposure.
• Comparison of agent assessments against human expert intuition in case studies.

Results & findings

• Embeddings effectively cluster physical risk sentences separately from general financial topics like 'revenue' and 'loss'.
• Sentiment analysis reveals that most physical risk disclosures are neutral, necessitating deeper materiality assessment.
• LLM agent correctly identified an $8.7 million write-off as material for a firm with weak financials but non-material for a stronger firm.
• Agent recognized long-term wildfire trends as material for a winemaker in fire-prone areas due to operational vulnerability.
• Agent accounted for geographic and operational context, such as water needs for drilling, to assess drought risk.
• Traditional keyword methods miss risks described indirectly, as shown in Box 1 examples.
• LLM agents provide a scalable way to augment human expert judgment in risk assessment.

Data & experiments

• Data source: SEC 10-K and 8-K filings from S&P Global.
• Extraction: Used OpenAI text-embedding-3-small to identify sentences semantically related to physical risks (drought, flood, etc.).
• Sentiment: Applied ProntoNLP to classify sentiment of extracted sentences.
• Materiality Assessment: LLM agent prompted with disclosure text and company financials (assets, revenue, EBITDA, etc.).
• Case Studies: Analyzed specific disclosures from Box 3 and Table 1 to validate agent performance.
• Visualization: Used t-SNE to visualize embedding clusters of physical risks vs. general financial topics.

Limitations & caveats

• Materiality assessment is inherently subjective and depends on user priorities.
• LLM agent performance relies on the accuracy and completeness of external financial data provided.
• The study is a case study and may not generalize to all industries or risk types.
• Dependence on specific models (OpenAI, ProntoNLP) may limit reproducibility.
• Final judgment still requires human expertise, limiting full automation.

Curator notes

Connects textual disclosures to measurable risk analytics.