The List / Reports, Filings, Accounting, and Risk / paper
The Structure of Financial Equity Research Reports
Abstract
This research dissects financial equity research reports (ERRs) by systematically mapping their content into categories. There is insufficient empirical analysis of the questions answered in ERRs. In particular, it is not understood how frequently certain information appears, what information is considered essential, and what information requires human judgment to distill into an ERR. The study analyzes 72 ERRs sentence-by-sentence, classifying their 4964 sentences into 169 unique question archetypes. We did not predefine the questions but derived them solely from the statements in the ERRs. This approach provides an unbiased view of the content of the observed ERRs. Subsequently, we used public corporate reports to classify the questions' potential for automation. Answers were labeled “text-extractable” if the answers to the question were accessible in corporate reports. 75.15% of the questions in ERRs can be automated using text extraction from text sources. Those automatable questions consist of 51.91% text-extractable (suited to processing by large language models, LLMs) and 24.24 % database-extractable questions. Only 24.85% of questions require human judgment to answer. We empirically validate, using Llama-3-70B and GPT-4-turbo-2024-04-09, that recent advances in language generation and information extraction enable the automation of approximately 80 % of the statements in ERRs. Surprisingly, the models complement each other's strengths and weaknesses well, indicating strong ensemble potential. The research confirms that the current writing process of ERRs can likely benefit from additional automation, improving quality and efficiency. The research thus allows us to quantify the potential impacts of introducing large language models in the ERR writing process. The full question list, including the archetypes and their frequency, are available online (janspoerer.github.io/pop-spoerer-2025-financial-report-data).
In one sentence
Summary
The paper addresses the lack of empirical analysis on the content structure of financial equity research reports (ERRs) and the specific questions analysts answer. It aims to quantify the portion of ERR writing that can be automated by large language models versus human judgment. The authors argue that understanding the frequency and extractability of information is crucial for designing automated systems that can assist or replace parts of the analyst workflow, particularly given the known biases and inefficiencies in traditional human-driven reporting. The methodology involves a manual, sentence-by-sentence analysis of 72 ERRs from 23 providers, deriving 169 unique question archetypes without pre-defined categories. These questions are classified by extractability (text-extractable, database-extractable, or non-extractable) using public corporate reports and financial databases. To validate automation potential, the authors test Llama-3-70B and GPT-4-turbo on 200 example questions, measuring their ability to extract answers from annual reports. They also conduct expert interviews to verify the relevance of the question list and assess the role of human judgment in areas like management quality assessment. Results show that 75.15% of questions in ERRs are automatable, with 51.91% being text-extractable and 24.24% database-extractable. Only 24.85% require human judgment, primarily in the 'Analysis' category (recommendations, target prices). Empirical tests reveal that Llama-3-70B and GPT-4 complement each other well, with an ensemble achieving an 84% success rate on extractable questions. The study concludes that significant automation is feasible, especially for factual reporting, but highlights limitations such as the inability of current models to handle complex, multi-source synthesis and the potential for out-of-distribution questions not captured in the sample.
Research problem
The paper investigates the structural composition of equity research reports (ERRs) to determine how much of the content can be automated. It addresses the gap in understanding which specific questions analysts answer, how frequently they appear, and which require human judgment versus simple data extraction. The goal is to quantify the automation potential of ERR writing using modern LLMs and information extraction techniques.
Core contributions
- A comprehensive, unbiased classification of 169 question archetypes derived from 72 equity research reports.
- Quantification of automation potential, showing 75.15% of ERR questions are automatable via text or database extraction.
- Empirical validation of LLM performance (Llama-3-70B and GPT-4) in extracting financial data from annual reports.
- Demonstration of strong ensemble potential between different LLMs, reducing error rates to approximately 1%.
- Identification of the 'Analysis' category (recommendations, target prices) as the primary domain requiring human judgment.
Method
- Unbiased question derivation: Reading sentences and generating questions ad-hoc.
- Classification of questions by extractability and display modality.
- Validation of extractability using Llama-3-70B and GPT-4-turbo on annual reports.
- Ensemble evaluation: Combining outputs from multiple LLMs to mitigate individual errors.
- Expert interviews to assess the role of subjective judgment in analyst reports.
- Statistical analysis of question frequency and category distribution.
Results & findings
- 75.15% of questions in ERRs are automatable; only 24.85% require human judgment.
- 51.91% of questions are text-extractable, suitable for LLM processing.
- 24.24% of questions are database-extractable, requiring financial data APIs.
- Llama-3-70B and GPT-4-turbo have complementary strengths, with minimal overlap in errors.
- Ensemble of Llama-3 and GPT-4 achieves 84% correctness on extractable questions.
- The 'Analysis' category (recommendations, target prices) is non-extractable and requires human insight.
- Financials and Product categories have high automation potential (70-73%).
- Market and Company categories have lower automation potential due to diverse data sources.
Data & experiments
- Dataset: 72 equity research reports from 23 providers (2018-2023), totaling 4,964 sentences.
- Question Derivation: Manual sentence-by-sentence annotation to create 169 unique question archetypes.
- Extractability Classification: Labeled questions as text-extractable, database-extractable, or non-extractable based on corporate reports and databases.
- LLM Evaluation: Tested Llama-3-70B and GPT-4-turbo on 200 example questions using annual reports as context.
- Expert Validation: Conducted interviews with 10 financial analysts to verify question relevance and human judgment requirements.
Limitations & caveats
- Sample size of 72 reports may not capture all question types across all providers.
- Frequency of questions does not necessarily correlate with their importance.
- Out-of-distribution questions may exist that were not captured in the analysis.
- The study focuses on text extraction and does not evaluate full report generation.
- Human judgment in areas like management quality is hard to quantify and automate.
- Market and Company categories require diverse data sources, limiting automation potential.
Curator notes
Important for decomposing report generation into extraction, database lookup, and human judgment.