TAT-LLM: A Specialized Language Model for Discrete Reasoning over Financial Tabular and Textual Data

In one sentence

Summary

The paper addresses the challenge of discrete reasoning over hybrid tabular and textual financial data, such as SEC filings, where models must perform arithmetic, comparison, and counting tasks. The authors argue that while large online LLMs like GPT-4 are capable, they pose cost, latency, and data security risks, motivating the specialization of smaller, open-source models for practical deployment in finance. The core problem is enabling these smaller models to execute multi-step inference reliably without the hallucination errors common in end-to-end generation. To solve this, the authors propose a Step-wise Pipeline consisting of three distinct stages: an Extractor to identify relevant evidence from tables and text, a Reasoner to generate the logical equation or rule, and an Executor to compute the final answer. They fine-tune LLaMA 2 (7B, 13B, 70B) using Low-Rank Adaptation (LoRA) on training data automatically generated from FinQA, TAT-QA, and TAT-DQA datasets. A key innovation is the External Executor, a post-processing module that evaluates the Reasoner's output to ensure mathematical correctness, addressing the model's weakness in calculation. The model outputs are structured as markdown tables to facilitate this pipeline. Experimental results demonstrate that TAT-LLM (7B) outperforms GPT-4 and all previous fine-tuned baselines on FinQA, TAT-QA, and TAT-DQA benchmarks. The 70B variant shows even greater gains, significantly closing the gap to human expert performance. The study highlights that the Step-wise Pipeline combined with the External Executor is critical for accuracy, particularly for arithmetic and counting tasks. However, the model is limited by input sequence length, making it unsuitable for very long documents (>100 pages), and it struggles with evidence extraction when financial terminology is complex or ambiguous.

Research problem

The paper tackles the problem of Question Answering (QA) over hybrid tabular and textual financial data, which requires discrete reasoning capabilities such as arithmetic calculation, comparison, and counting. It specifically addresses the limitations of end-to-end LLM approaches and the practical constraints (cost, latency, privacy) of using massive online models like GPT-4 for financial document analysis.

Core contributions

• Abstracts a Step-wise Pipeline (Extractor, Reasoner, Executor) for discrete reasoning over tabular-textual data.
• Develops TAT-LLM by fine-tuning LLaMA 2 (7B-70B) using LoRA on automatically generated training data.
• Introduces an External Executor module to ensure mathematical accuracy of the final answer.
• Demonstrates that a 7B specialized model outperforms GPT-4 and prior SOTA models on FinQA, TAT-QA, and TAT-DQA.
• Provides detailed error analysis showing that evidence extraction is the primary bottleneck for smaller models.

Method

• Step-wise Pipeline: Decomposes QA into Evidence Extraction, Logic/Equation Generation, and Execution.
• Model Base: LLaMA 2 fine-tuned with Low-Rank Adaptation (LoRA).
• Data Construction: Transforms existing QA datasets into instruction-response pairs formatted as structured tables.
• External Executor: A rule-based module that parses the Reasoner's output to execute equations or validate logic.
• Evaluation: Uses Exact Match and F1 scores, with scale prediction handled separately for numerical answers.
• Training: Supervised fine-tuning on combined datasets from FinQA, TAT-QA, and TAT-DQA.

Results & findings

• TAT-LLM (7B) achieves 64.60% EM on FinQA, beating GPT-4 (63.91%).
• On TAT-QA, TAT-LLM (7B) reaches 74.56% EM and 82.88% F1, surpassing GPT-4.
• On TAT-DQA, TAT-LLM (7B) achieves 69.45% EM, outperforming GPT-4 (64.46%).
• The External Executor provides significant gains, especially for smaller models (e.g., +16.66% EM on FinQA for 7B).
• Larger models (70B) benefit less from the External Executor as their internal reasoning is more robust.
• Arithmetic and Counting questions show the largest performance improvements over baselines.
• Error analysis reveals that 48% of errors stem from the Extractor misidentifying relevant evidence.

Data & experiments

• Datasets: FinQA, TAT-QA, and TAT-DQA, which contain financial reports with tables and text.
• Baselines: Compared against fine-tuned models (e.g., DyRRen, MVGE) and zero-shot LLMs (GPT-3.5, GPT-4, LLaMA 2, Vicuna).
• Metrics: Exact Match (EM) and numeracy-focused F1 scores.
• Experiments: Evaluated TAT-LLM variants (7B, 13B, 70B) with and without the External Executor.
• Ablation studies on fine-tuning strategies (End-to-end vs. Step-wise) and question types (Arithmetic, Counting, Span).

Limitations & caveats

• Limited by the maximum input sequence length of LLaMA 2 (4096 tokens), restricting use on very long documents.
• Performance drops significantly if the model fails to extract the correct evidence in the first step.
• May not generalize well to pure textual documents without tabular data.
• The External Executor relies on the Reasoner producing a valid equation, which can fail for complex logic.
• Training data is derived from existing datasets, potentially limiting diversity in question types.

Curator notes

Discovery status: round1_promoted_seed_for_round2. Source links: BloombergGPT: A Large Language Model for Finance; ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering; FinQA: A Dataset of Numerical Reasoning over Financial Data; FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models; FinVis-GPT: A Multimodal Large Language Model for Financial Chart Analysis; Instruct-FinGPT: Financial Sentiment Analysis by Instruction Tuning of General-Purpose Large Language Models; InvestLM: A Large Language