The List / Financial QA, Reasoning, and Table Understanding / paper
MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data
Abstract
Numerical reasoning over hybrid data containing both textual and tabular content (e.g., financial reports) has recently attracted much attention in the NLP community. However, existing question answering (QA) benchmarks over hybrid data only include a single flat table in each document and thus lack examples of multi-step numerical reasoning across multiple hierarchical tables. To facilitate data analytical progress, we construct a new large-scale benchmark, MultiHiertt, with QA pairs over Multi Hierarchical Tabular and Textual data. MultiHiertt is built from a wealth of financial reports and has the following unique characteristics: 1) each document contain multiple tables and longer unstructured texts; 2) most of tables contained are hierarchical; 3) the reasoning process required for each question is more complex and challenging than existing benchmarks; and 4) fine-grained annotations of reasoning processes and supporting facts are provided to reveal complex numerical reasoning. We further introduce a novel QA model termed MT2Net, which first applies facts retrieving to extract relevant supporting facts from both tables and text and then uses a reasoning module to perform symbolic reasoning over retrieved facts. We conduct comprehensive experiments on various baselines. The experimental results show that MultiHiertt presents a strong challenge for existing baselines whose results lag far behind the performance of human experts. The dataset and code are publicly available at https://github.com/psunlpgroup/MultiHiertt.
In one sentence
Summary
This paper addresses the limitation of existing financial question-answering benchmarks, which typically rely on single flat tables, by introducing MultiHiertt. This new dataset is constructed from S&P 500 annual reports and features documents containing multiple hierarchical tables and long unstructured texts. The primary research problem is enabling systems to perform complex, multi-step numerical reasoning that requires integrating information across these hierarchical structures and textual paragraphs, a task that mirrors the analytical work of financial professionals. The dataset includes 10,440 expert-annotated question-answer pairs with fine-grained reasoning programs and supporting facts, ensuring high quality and diversity in reasoning complexity. To tackle this challenge, the authors propose MT2Net, a novel QA model consisting of a fact-retrieving module and a reasoning module. The retrieving module uses a BERT-based classifier to extract relevant supporting facts from both hierarchical tables and text, preserving the table's structural hierarchy by flattening cells with their row and column headers. The reasoning module then employs either a program generator or a span selector to compute the final answer. Experiments are conducted on MultiHiertt using various baselines, including FinQANet, TAGOP, and Longformer, with evaluation metrics including Exact Match and F1 scores. The study also includes a human performance baseline to assess the difficulty of the task. The results demonstrate that MT2Net outperforms all existing baselines, achieving an F1 score of 38.43% on the test set, which is a significant improvement over previous models. However, this performance still lags far behind human experts, who achieved an F1 score of 87.03%. The analysis reveals that models struggle particularly with questions requiring reasoning across multiple tables, multi-step reasoning, and complex hierarchical structures. The paper highlights that current models fail to integrate supporting facts correctly and lack external financial knowledge. These findings indicate that MultiHiertt presents a strong challenge for current NLP models and underscores the need for more advanced table-encoding methods and multi-table reasoning mechanisms in financial AI systems.
Research problem
Existing financial QA benchmarks lack examples of multi-step numerical reasoning across multiple hierarchical tables and long texts. The paper aims to create a benchmark that reflects the complexity of real-world financial report analysis, where professionals must integrate information from multiple sources and hierarchical structures to answer complex questions.
Core contributions
- Introduction of MultiHiertt, a large-scale dataset with 10,440 QA pairs over multi-hierarchical tabular and textual data from S&P 500 reports.
- Development of MT2Net, a novel QA model with a fact-retrieving module and a reasoning module designed for hierarchical table understanding.
- Demonstration that current models significantly underperform human experts, highlighting the difficulty of the task.
- Comprehensive analysis of model errors, identifying key failure modes such as multi-table reasoning and lack of domain knowledge.
Method
- Fact Retrieval: BERT-based classifier to retrieve top-n supporting facts from tables and text.
- Table Encoding: Flattening hierarchical tables by including row and column headers in cell representations.
- Reasoning Module: LSTM-based program generator for arithmetic operations and span selector for text extraction.
- Joint Probability: Combining type prediction and answer probability for final output.
- Evaluation: EM and F1 scores, with error analysis categorizing mistakes.
Results & findings
- MT2Net achieves 38.43% F1 on the test set, outperforming all baselines.
- Human experts achieve 87.03% F1, showing a large performance gap.
- Models struggle with questions requiring reasoning across multiple tables.
- Performance drops significantly as the number of reasoning steps increases.
- 64% of errors are due to failure to integrate supporting facts correctly.
- Hierarchical table structure is crucial; ignoring it leads to poor performance.
Data & experiments
- Dataset: MultiHiertt, derived from FinTabNet, containing 2,513 documents with multiple hierarchical tables and long texts.
- Annotation: 10,440 QA pairs with fine-grained reasoning programs and supporting facts, annotated by finance experts.
- Baselines: FinQANet, TAGOP, Longformer, TAPAS, NumNet+, and Seq2Prog.
- Metrics: Exact Match (EM) and F1 score.
- Human Performance: Evaluated on a sample of the test set to establish a performance ceiling.
Limitations & caveats
- Models still perform significantly worse than human experts.
- Difficulty with multi-table reasoning and complex hierarchical structures.
- Lack of external financial knowledge in current models.
- Performance degrades with increased reasoning steps.
- Potential bias in annotation process, though mitigated by quality control.
Curator notes
Important for filings with multiple tables and hierarchical structure.