The List / Financial QA, Reasoning, and Table Understanding / paper
TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance
Abstract
Hybrid data combining both tabular and textual content (e.g., financial reports) are quite pervasive in the real world. However, Question Answering (QA) over such hybrid data is largely neglected in existing research. In this work, we extract samples from real financial reports to build a new large-scale QA dataset containing both Tabular And Textual data, named TAT-QA, where numerical reasoning is usually required to infer the answer, such as addition, subtraction, multiplication, division, counting, comparison/sorting, and the compositions. We further propose a novel QA model termed TAGOP, which is capable of reasoning over both tables and text. It adopts sequence tagging to extract relevant cells from the table along with relevant spans from the text to infer their semantics, and then applies symbolic reasoning over them with a set of aggregation operators to arrive at the final answer. TAGOP achieves 58.0% inF1, which is an 11.1% absolute increase over the previous best baseline model, according to our experiments on TAT-QA. But this result still lags far behind performance of expert human, i.e.90.8% in F1. It is demonstrated that our TAT-QA is very challenging and can serve as a benchmark for training and testing powerful QA models that address hybrid form data.
In one sentence
Summary
The paper addresses the challenge of Question Answering (QA) over hybrid data, specifically the combination of tabular and textual content found in real-world financial reports. Existing QA systems typically focus on unstructured text or structured knowledge bases, neglecting the complex interdependencies and numerical reasoning required when tables and accompanying paragraphs are semantically linked. The authors argue that this hybrid form is pervasive in finance and requires models to extract information from both modalities and perform arithmetic operations to derive answers. To tackle this, the authors construct TAT-QA, a large-scale dataset containing 16,552 questions derived from 2,757 hybrid contexts extracted from 182 real financial reports. The dataset includes questions requiring various numerical reasoning types such as addition, subtraction, division, and counting, with annotated derivations and scales. They propose TAGOP, a model that uses a RoBERTa-based encoder for sequence tagging to extract evidence cells and spans, followed by an operator classifier to select from ten symbolic aggregation operators. The model also includes a scale prediction component to handle units like millions or billions. Experiments compare TAGOP against textual, tabular, and hybrid baselines like BERT-RC, TaPas, and HyBrider. TAGOP achieves 58.0% F1 on the test set, an 11.1% absolute improvement over the best baseline, demonstrating the effectiveness of symbolic reasoning over hybrid data. However, this performance still lags significantly behind human experts, who achieve 90.8% F1. Error analysis reveals that 84% of errors stem from incorrect evidence extraction, highlighting the difficulty of aligning table and text. The paper concludes that TAT-QA is a challenging benchmark that exposes current limitations in numerical reasoning and hybrid data understanding, serving as a foundation for future research in financial QA.
Research problem
The paper addresses the lack of research on Question Answering (QA) over hybrid data, which combines tabular and textual content. It highlights that existing QA systems focus on either unstructured text or structured tables, ignoring the complex numerical reasoning and semantic linking required in real-world scenarios like financial reports. The specific problem is to develop a benchmark and model capable of answering questions that require extracting and aggregating information from both tables and associated text paragraphs.
Core contributions
- Introduction of TAT-QA, a large-scale QA dataset with 16,552 questions from 182 financial reports, focusing on hybrid tabular-textual contexts.
- Development of TAGOP, a novel QA model that combines sequence tagging for evidence extraction with symbolic aggregation operators for numerical reasoning.
- Incorporation of scale prediction to handle financial units (e.g., million, billion) which are often implicit or distributed across headers and text.
- Comprehensive evaluation showing TAGOP outperforms textual, tabular, and hybrid baselines, while identifying significant gaps compared to human performance.
- Detailed error analysis categorizing failures into evidence extraction, calculation, and scale prediction, providing insights for future model improvements.
Method
- Sequence Tagging: Uses RoBERTa encoder with Inside-Outside (IO) tagging to extract relevant table cells and text spans.
- Operator Classification: A multi-class classifier predicts one of ten aggregation operators (e.g., Sum, Difference, Change Ratio) based on the [CLS] token.
- Symbolic Reasoning: Applies the predicted operator to the extracted numerical evidence to compute the answer.
- Scale Prediction: A separate classifier predicts the magnitude scale (None, Thousand, Million, Billion, Percent) using concatenated representations of question, table, and text.
- Number Order Classification: An auxiliary classifier determines the order of operands for order-sensitive operators like Division and Difference.
- Loss Function: Combines negative log-likelihood losses for tagging, operator, scale, and order prediction tasks.
Results & findings
- TAGOP achieves 58.0% F1 on the TAT-QA test set, an 11.1% absolute increase over the best baseline (NumNet+ V2 at 46.9%).
- Human experts achieve 90.8% F1, indicating the task remains significantly challenging for current AI models.
- TAGOP outperforms tabular-only (TaPas) and hybrid-only (HyBrider) baselines, demonstrating the value of joint reasoning.
- Error analysis shows 55% of errors are due to wrong evidence extraction and 29% to missing evidence.
- Only 9% of errors are due to wrong calculation, suggesting evidence retrieval is the primary bottleneck.
- Scale prediction accuracy is high (90.1% on test set), but scale errors still contribute to final answer mismatches.
- Arithmetic questions are harder than span extraction questions, with TAGOP performing relatively worse on them.
- The dataset includes diverse reasoning types: 19.69% composition, 16.17% subtraction, and 11.94% set of spans.
Data & experiments
- Dataset: TAT-QA contains 2,757 hybrid contexts (tables + paragraphs) from 182 real financial reports, with 16,552 annotated question-answer pairs.
- Annotation: Questions require numerical reasoning (addition, subtraction, etc.) and include derivations; annotators have financial backgrounds.
- Baselines: Evaluated against textual models (BERT-RC, NumNet+ V2), tabular model (TaPas), and hybrid model (HyBrider).
- Metrics: Exact Match (EM) and numeracy-focused F1 score, adjusted to respect positive/negative signs and scales.
- Human Baseline: Human experts achieved 90.8% F1 on a sample of the test set, providing a performance ceiling.
- Error Analysis: Manual inspection of 100 error cases to categorize failure modes like wrong evidence or unsupported calculations.
Limitations & caveats
- The dataset is limited to financial reports, which may not generalize to other hybrid data domains without adaptation.
- The TAGOP model relies on predefined operators, which may not cover all possible reasoning patterns in complex financial queries.
- Human performance is still significantly higher than AI, indicating that current models are far from fully solving the problem.
- The dataset construction involves manual annotation, which is costly and may introduce subjectivity or errors.
- The model's performance on 'unsupported calculations' (4% of errors) suggests limitations in handling novel or complex arithmetic not covered by the ten operators.
- The paper notes that some questions implicitly require domain knowledge (e.g., gross profit margin formula), which the model may not fully capture.
Curator notes
Tests operations such as addition, subtraction, division, sorting, and counting.