ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering

In one sentence

Summary

The paper addresses the challenge of complex numerical reasoning in financial natural language processing, specifically focusing on conversational question answering over financial reports. The authors propose ConvFinQA, a new dataset comprising 3,892 conversations and 14,115 questions derived from decomposing and concatenating multi-hop questions from the FinQA dataset. This construction simulates realistic investor-analyst dialogues where later questions depend on previous answers, creating long-range reasoning chains that test a model's ability to maintain context and perform sequential calculations across text and tables. The dataset includes both simple conversations from single question decomposition and hybrid conversations integrating two distinct aspects of a report, requiring models to navigate cross-dependencies and varying levels of numerical complexity. Experiments evaluate two primary approaches: neural symbolic models and prompting-based large language models. The neural symbolic approach utilizes FinQANet, a pipeline combining a retriever for supporting facts with a generator for reasoning programs, tested with BERT and RoBERTa encoders. The prompting-based approach employs GPT-3 (text-davinci-002) using few-shot learning with various prompt formats, including answer-only, program generation in original and normal DSLs, and Chain-of-Thought reasoning. The evaluation metrics are execution accuracy and program accuracy. Results show that the best neural symbolic model (FinQANet with RoBERTa-large) achieves 68.90% execution accuracy, while the best prompting method (Program-normal) reaches only 48.85%. Both methods struggle significantly with later conversation turns and hybrid conversations, where dependency distances are longer and context management is more complex. The findings highlight a substantial gap between current AI systems and human expert performance, which stands at 89.44% execution accuracy. Neural symbolic models excel at number selection but fail at complex program generation due to lack of domain knowledge and difficulty with long reasoning chains. GPT-3 performs poorly on the conversational paradigm, often mimicking exemplars rather than understanding the task, and struggles with references to previous context. The paper concludes that while LMs are powerful, they currently lack the robust reasoning capabilities required for real-world financial analysis tasks involving multi-turn, complex numerical dependencies. Limitations include the dataset's construction method, which may not cover all real-world conversational patterns, and the exclusive focus on GPT-3 for prompting experiments, leaving open questions about other models' performance in this specific domain.

Research problem

The paper investigates the ability of current NLP models to perform complex, multi-turn numerical reasoning in a financial context. Specifically, it addresses the challenge of conversational question answering over financial reports, where questions form chains of dependencies requiring sequential calculations and context retention. The core problem is determining whether neural symbolic methods or prompting-based LLMs are better suited for this realistic, domain-specific reasoning task.

Core contributions

• Introduction of ConvFinQA, a large-scale dataset with 3,892 conversations and 14,115 questions designed for conversational financial numerical reasoning.
• A novel dataset construction framework that simulates realistic conversation flows by decomposing and concatenating multi-hop questions from FinQA.
• Comprehensive benchmarking of neural symbolic models (FinQANet) and prompting-based LLMs (GPT-3) on the new dataset.
• Detailed analysis of model failures, highlighting difficulties with long-range dependencies, domain knowledge gaps, and paradigm understanding in LLMs.
• Public release of the dataset and code to facilitate future research in financial reasoning and conversational QA.

Method

• Dataset construction via decomposition of multi-hop FinQA questions into conversation turns.
• Concatenation of decomposed questions to form hybrid conversations.
• Expert annotators compose natural language questions from reasoning skeletons.
• FinQANet pipeline: Retriever for supporting facts, Generator for reasoning programs.
• GPT-3 prompting: Few-shot learning with various prompt formats (answer-only, program, CoT).
• Evaluation using execution accuracy and program accuracy.
• Analysis of performance by question type, conversation type, and turn number.

Results & findings

• ConvFinQA dataset size: 3,892 conversations, 14,115 questions.
• Human expert execution accuracy: 89.44%.
• Best neural symbolic model (FinQANet-RoBERTa-large) execution accuracy: 68.90%.
• Best prompting method (GPT-3 Program-normal) execution accuracy: 48.85%.
• Hybrid conversations are significantly harder than simple conversations.
• Performance degrades significantly in later conversation turns due to longer dependency chains.
• GPT-3 struggles with conversational context references, often ignoring previous turns.
• Neural symbolic models suffer from lack of domain knowledge and difficulty with long reasoning chains.

Data & experiments

• ConvFinQA dataset: 3,892 conversations, 14,115 questions, derived from financial reports using decomposition and concatenation of FinQA questions.
• Two conversation types: Type I (simple, from single question decomposition) and Type II (hybrid, from two questions).
• Neural symbolic experiments: FinQANet with BERT/RoBERTa encoders, using a retriever-generator pipeline.
• Prompting experiments: GPT-3 (text-davinci-002) with various few-shot prompts (answer-only, program-original, program-normal, CoT).
• Evaluation metrics: Execution accuracy and program accuracy.
• Human performance baseline: Expert annotators achieved 89.44% execution accuracy.

Limitations & caveats

• Dataset construction relies on decomposition of existing FinQA questions, which may not cover all real-world conversational patterns.
• Prompting experiments limited to GPT-3, leaving other models untested.
• Retrieval performance for GPT-3 was lower than trained retrievers, potentially affecting results.
• The dataset may not fully capture the complexity of real investor-analyst interactions.
• Models still significantly lag behind human experts, indicating room for improvement.

Curator notes

Extends financial reasoning into multi-turn conversation.