PyFi: Toward Pyramid-like Financial Image Understanding for VLMs via Adversarial Agents

In one sentence

Summary

The paper addresses the scarcity of high-quality, expert-level visual reasoning data for financial vision-language models (VLMs), noting that existing benchmarks rely on costly human annotation or hallucinated synthetic data. It proposes PyFi, a framework that structures financial image understanding into a six-level pyramid ranging from basic perception to complex decision support, enabling progressive reasoning through interconnected question chains. This hierarchical approach aims to provide interpretable, step-wise guidance for VLMs to solve intricate financial problems that require deep domain expertise and visual analysis. The core contribution is PyFi-600K, a dataset of 600,000 samples synthesized automatically using PyFi-adv, a multi-agent adversarial mechanism based on Monte Carlo Tree Search. A challenger agent generates increasingly difficult questions while a solver agent attempts to answer them, creating hierarchical question chains with reward scores for process supervision. The authors evaluate 15 pre-trained VLMs, revealing a sharp accuracy drop from 71.80% at the perception level to 32.95% at the decision level. They then fine-tune Qwen2.5-VL-3B and 7B models on these chains, demonstrating that supervised fine-tuning with chain-of-thought annotations significantly enhances the models' ability to decompose and solve complex financial visual tasks. Results show that fine-tuning yields average accuracy improvements of 19.52% for the 3B model and 8.06% for the 7B model, with the smaller model benefiting disproportionately from the chain-of-thought training. The study highlights that errors in calculation analysis and data extraction are primary causes of failure in higher-level decision support. While the framework effectively improves interpretability and performance on financial charts and infographics, the reliance on synthetic data generation introduces potential biases, and the current scope is limited to visual reasoning rather than broader market prediction or trading execution.

Research problem

The paper addresses the lack of scalable, high-quality datasets for training and evaluating vision-language models (VLMs) in the financial domain. Specifically, it tackles the difficulty of creating hierarchical, expert-level visual reasoning data that supports step-by-step problem solving, which is essential for complex financial decision-making tasks involving charts, infographics, and financial documents.

Core contributions

• Proposes PyFi, a framework for pyramid-like financial image understanding that structures reasoning into six hierarchical levels.
• Introduces PyFi-600K, a large-scale dataset of 600,000 samples organized into reasoning pyramids with interconnected question chains.
• Develops PyFi-adv, a multi-agent adversarial synthesis mechanism using Monte Carlo Tree Search to automatically generate high-quality training data without human annotation.
• Provides a comprehensive evaluation of 15 state-of-the-art VLMs, revealing significant performance gaps in complex financial visual reasoning.
• Demonstrates that fine-tuning VLMs on pyramid-structured question chains significantly improves accuracy and interpretability in financial decision support.

Method

• Pyramid structure defining six levels of financial visual cognition: Perception, Data Extraction, Calculation Analysis, Pattern Recognition, Logical Reasoning, and Decision Support.
• Sample chains linking questions across levels to demonstrate progressive reasoning.
• PyFi-adv mechanism using a challenger agent to generate difficult questions and a solver agent to answer them.
• Monte Carlo Tree Search (MCTS) to guide the adversarial interaction and explore diverse question paths.
• Backpropagation of reward scores to individual samples in the chain for process supervision.
• Supervised Fine-Tuning (SFT) with LoRA on Qwen2.5-VL models using chain-of-thought annotations.
• Data leakage mitigation by excluding samples answerable without visual input.

Results & findings

• Pre-trained VLMs show an average accuracy drop from 71.80% at Level 1 (Perception) to 32.95% at Level 6 (Decision Support).
• GLM-4.5V achieves the highest overall accuracy of 74.75% among pre-trained models.
• Fine-tuning Qwen2.5-VL-3B with chain-of-thought data improves accuracy by 19.52 percentage points.
• Fine-tuning Qwen2.5-VL-7B with chain-of-thought data improves accuracy by 8.06 percentage points.
• The 3B model fine-tuned with PyFi surpasses the 72B base model in Pattern Recognition and Logical Reasoning tasks.
• Addressing a Level-6 question typically requires correctly answering an average of 10.48 sub-questions from lower levels.
• Calculation analysis errors are identified as a primary cause of failure in financial decision-making tasks.

Data & experiments

• PyFi-600K dataset containing 603,859 samples across 17 financial themes and 11 image types.
• Evaluation of 15 pre-trained VLMs including GPT-4.1, Claude-opus, GLM-4.5V, and Qwen2.5-VL variants.
• Fine-tuning experiments on Qwen2.5-VL-3B and Qwen2.5-VL-7B using supervised fine-tuning with and without chain-of-thought annotations.
• Data leakage mitigation by filtering samples that can be answered without visual input.
• Analysis of performance across six capability levels and five complexity degrees.

Limitations & caveats

• The dataset is synthetically generated, which may introduce biases or hallucinations not present in real-world data.
• The framework is limited to visual reasoning tasks and does not address broader financial text analysis or market prediction.
• The effectiveness of fine-tuning varies by model size, with smaller models benefiting more significantly.
• The adversarial synthesis process is computationally intensive, though parallelizable.
• The evaluation focuses on specific financial image types, potentially limiting generalizability to other domains.

Curator notes

Discovery status: round3_promoted_seed_for_round4. Source links: BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMs; FinBERT : A Large Language Model for Extracting Information from Financial Text†; FinVis-GPT: A Multimodal Large Language Model for Financial Chart Analysis; HedgeAgents: A Balanced-aware Multi-agent Financial Trading System; PIXIU: A Comprehensive Benchmark, Instruction Dataset and Large Language Model for Finance