EDINET-Bench: Evaluating LLMs on Complex Financial Tasks using Japanese Financial Statements

In one sentence

Summary

This paper addresses the lack of expert-level financial benchmarks by introducing EDINET-Bench, an open-source dataset for evaluating Large Language Models on complex Japanese financial tasks. The benchmark focuses on three challenging classification problems: accounting fraud detection, earnings forecasting, and industry prediction. These tasks require models to process entire annual reports, integrating information across multiple tables and textual sections, demanding reasoning capabilities that are difficult even for human professionals. The study positions this work as a critical step toward benchmarks that reflect the high-stakes, multi-modal nature of real-world financial analysis, moving beyond simple knowledge retrieval or basic numerical QA. The dataset is constructed from ten years of annual reports filed via Japan’s EDINET platform, comprising approximately 40,000 documents. The authors developed edinet2dataset to parse these reports into structured components including balance sheets, cash flow statements, profit and loss statements, and textual narratives. For fraud detection, they identified fraudulent cases by analyzing amendment reasons using LLMs and manual review, resulting in a balanced binary classification dataset. Earnings forecasting was created by predicting year-over-year profit changes, while industry prediction used a consolidated 16-category classification scheme. Experiments evaluated zero-shot performance of closed-source models (GPT-4o, Claude 3.5/3.7, GPT-5) and open-weight models (Llama 3.3, DeepSeek-V3/R1) against classical baselines like Logistic Regression and Random Forest, using ROC-AUC, MCC, and accuracy metrics. Results indicate that even state-of-the-art LLMs struggle significantly, performing only marginally better than logistic regression in fraud detection and earnings forecasting. For instance, GPT-5 achieved a ROC-AUC of 0.65 in earnings forecasting, while Random Forest outperformed all LLMs in fraud detection. Incorporating textual information improved fraud detection but not earnings forecasting, suggesting different reliance on narrative signals. The study also assessed contamination risks, finding minimal impact through company name prediction tests. The authors conclude that simply providing reports is insufficient, advocating for benchmark frameworks with richer scaffolding, such as realistic simulations and task-specific reasoning support, to better emulate professional financial environments.

Research problem

The financial domain lacks benchmarks that test expert-level reasoning, with existing datasets focusing on simple QA or extraction. This paper addresses the need for a challenging benchmark that requires integrating tabular and textual data from full annual reports to perform complex tasks like fraud detection and earnings forecasting, which are critical for real-world financial analysis but underrepresented in current LLM evaluations.

Core contributions

• Introduction of EDINET-Bench, the first open dataset for accounting fraud detection using Japanese financial reports.
• Development of edinet2dataset, a toolkit for downloading and parsing EDINET reports into structured financial components.
• Creation of three expert-level tasks: fraud detection, earnings forecasting, and industry prediction, requiring multi-modal reasoning.
• Comprehensive evaluation of frontier LLMs against classical baselines, revealing significant performance gaps in complex financial reasoning.
• Analysis of contamination risks and feature importance, providing insights into model biases and limitations in financial document understanding.

Method

• Data extraction using edinet2dataset to parse EDINET reports into BS, CF, PL, Summary, and Text components.
• Fraud label generation via LLM analysis of amendment reasons, followed by manual review for validation.
• Earnings labels derived from year-over-year changes in 'Profit Attributable to Owners of the Parent Company'.
• Industry labels consolidated from 33 SICC sectors to 16 broader categories based on MIC guidelines.
• Zero-shot evaluation of LLMs with prompts requesting prediction, probability, and reasoning.
• Input format variations tested: Summary only, Summary+BS+CF+PL, and all components including Text.
• Classical baselines (Logistic Regression, Random Forest, XGBoost) trained on the same splits for comparison.
• Contamination assessment via company name prediction and year-wise performance analysis.

Results & findings

• State-of-the-art LLMs perform only marginally better than logistic regression in fraud detection and earnings forecasting.
• GPT-5 achieved a ROC-AUC of 0.65 in earnings forecasting, aligning with previous low performance reports.
• Random Forest outperformed all LLMs in fraud detection, achieving the highest ROC-AUC and MCC scores.
• Incorporating textual information significantly improved fraud detection performance but had no effect on earnings forecasting.
• Claude 3.5 Sonnet achieved the best industry prediction accuracy of 0.41 with tabular data, significantly above random guessing.
• Feature importance analysis revealed logistic regression bias towards larger companies in fraud detection.
• Contamination tests showed minimal impact, with company name prediction accuracy below 5% for all models.
• Models exhibited bias towards reputable auditors in fraud detection reasoning, highlighting potential heuristic reliance.

Data & experiments

• Dataset constructed from 10 years of EDINET annual reports, covering ~40,000 documents from Japanese listed companies.
• Fraud detection dataset built from amended reports, with labels verified by LLMs and manual review, resulting in ~1,000 instances.
• Earnings forecasting dataset created by predicting year-over-year profit changes for 1,000 companies.
• Industry prediction dataset uses a consolidated 16-category scheme, with ~1,000 instances across 16 industries.
• Evaluation includes zero-shot testing of GPT-4o, Claude 3.5/3.7, GPT-5, Llama 3.3, DeepSeek-V3/R1, and classical baselines.
• Metrics include ROC-AUC, MCC for binary tasks, and accuracy for multi-class, with analysis of input format effects.

Limitations & caveats

• Dataset is limited to Japanese financial reports, potentially limiting generalizability to other markets.
• Fraud detection labels rely on amendment reasons, which may not capture all forms of accounting fraud.
• Zero-shot evaluation may not reflect performance with fine-tuning or specialized prompting strategies.
• Potential for test set contamination exists, though mitigated by contamination analysis.
• Textual information in annual reports may contain company names, which could leak industry labels in prediction tasks.
• Classical baselines were trained on the same data, which may not reflect real-world deployment scenarios.

Curator notes

Discovery status: report_analysis_focus_promoted. Source links: Financial Statement Analysis with Large Language Models