Can large language models effectively process and execute financial trading instructions?

In one sentence

Summary

This research addresses the challenge of integrating Large Language Models (LLMs) into financial trading systems by developing an intelligent trade order recognition pipeline. The primary goal is to enable the conversion of unstructured trade orders into a standard format, specifically JSON, to facilitate automated trade execution. The study aims to improve human-trader interaction with platforms while mitigating misinformation risks during the execution phase. By focusing on the initial step of order recognition, the authors seek to bridge the gap between natural language intent and structured system commands. The methodology involves creating a custom dataset of 500 trade order pieces to simulate real-world scenarios. Five state-of-the-art LLMs were evaluated using this dataset to assess their generative power and reliability in finance. The experimental design includes specific metrics for dataset reliability and model performance, focusing on syntactic validity, information completeness, and the frequency of clarifying questions. The system architecture features an agent collaboration model with user and trading agents, supported by carefully designed prompts for core tasks, instruction generation, and financial reasoning. Results indicate that while most models generate syntactically valid JSON at high rates (80%-99%) and initiate clarifying questions in nearly all incomplete cases (90%-100%), end-to-end accuracy remains low (6%-14%). A significant limitation is the tendency for models to over-interrogate, with 70%-80% of follow-ups being unnecessary, which raises interaction costs and information-exposure risks. Despite these accuracy issues, the research demonstrates the feasibility of integrating the pipeline with real-world trading systems, paving the way for practical deployment of LLM-based trade automation solutions.

Research problem

The paper addresses the difficulty of integrating Large Language Models (LLMs) into existing financial trading systems. Specifically, it tackles the challenge of converting unstructured natural language trade orders into standardized, executable formats. The research aims to improve the interaction between human traders and trading platforms while reducing the risk of misinformation during the trade execution process.

Core contributions

• Proposes an intelligent trade order recognition pipeline for converting natural language to standard JSON.
• Creates a custom dataset of 500 trade order pieces to simulate real-world trading scenarios.
• Designs specific metrics to assess dataset reliability and LLM generative power in finance.
• Evaluates five state-of-the-art LLMs on their ability to process and execute financial instructions.
• Demonstrates the feasibility of integrating the proposed pipeline with real-world trading systems.

Method

• Designed an intelligent trade order recognition pipeline.
• Created a dataset of 500 simulated trade orders.
• Evaluated five state-of-the-art LLMs on the dataset.
• Developed metrics for syntactic validity and information completeness.
• Implemented an agent collaboration architecture with user and trading agents.
• Designed specific prompts for core tasks, instruction generation, and reasoning.
• Assessed the frequency and necessity of clarifying questions.
• Tested integration feasibility with real-world trading systems.

Results & findings

• Most models generate syntactically valid JSON at high rates (80%-99%).
• Models initiate clarifying questions in nearly all incomplete cases (90%-100%).
• End-to-end accuracy remains low, ranging from 6% to 14%.
• Missing information is substantial, ranging from 12% to 66%.
• 70%-80% of model follow-up questions are unnecessary, raising interaction costs.
• Over-interrogation poses potential information-exposure risks.
• The pipeline is feasible for integration with real-world trading systems.

Data & experiments

• Developed a proprietary dataset of 500 trade order pieces simulating real-world scenarios.
• Evaluated five state-of-the-art Large Language Models (LLMs) on the custom dataset.
• Assessed syntactic validity, information completeness, and clarifying question frequency.
• Measured end-to-end accuracy of order recognition and execution readiness.
• Analyzed the cost and risk implications of unnecessary model follow-ups.

Limitations & caveats

• End-to-end accuracy is low (6%-14%), limiting practical utility.
• High rate of missing information (12%-66%) in generated orders.
• Excessive unnecessary clarifying questions increase interaction costs.
• Over-interrogation poses information-exposure risks.
• Dataset size (500 pieces) may be limited for robust generalization.
• Feasibility of integration is demonstrated but not fully validated in live trading.

Curator notes

Discovery status: institutional_trading_focus_promoted. Source links: large language models execution trading