LOB-Bench: Benchmarking Generative AI for Finance - an Application to Limit Order Book Data

In one sentence

Summary

The paper addresses the critical lack of standardized, quantitative evaluation metrics for generative models in financial market microstructure, specifically for limit order book (LOB) data. Existing methods rely on qualitative assessments of stylized facts or simple cross-entropy losses that fail to capture error accumulation during autoregressive sampling. The authors propose LOB-Bench, a Python-based framework that evaluates the distributional similarity between real and generated LOB data using both unconditional and conditional statistical metrics. This includes measuring distributional differences in key LOB statistics such as spread, order book volumes, order imbalance, and message inter-arrival times, alongside adversarial discriminator scores and market impact response functions. The framework is designed to be extensible and applicable to other high-dimensional financial time series. The study benchmarks four modern generative AI models against a traditional parametric baseline: an autoregressive state-space model (LOBS5), a conditional GAN (Coletta), and two RWKV variants. Experiments are conducted on high-frequency LOB data for Alphabet (GOOG) and Intel (INTC) stocks in the LOBSTER format. The evaluation pipeline computes L1 and Wasserstein-1 distances between real and generated distributions, assesses conditional dependencies (e.g., spread conditioned on time of day), and measures the divergence of errors over increasing prediction horizons to identify "model derailment." Additionally, the authors train a discriminator network to distinguish real from generated trajectories and evaluate the downstream utility of synthetic data by training a mid-price prediction classifier. Results indicate that the autoregressive LOBS5 model significantly outperforms traditional classes, achieving the lowest distributional errors and best reproduction of market impact curves, particularly for the small-tick GOOG stock. The baseline parametric model fails to capture complex dynamics, while RWKV models exhibit rapid error accumulation and distributional drift. The Coletta model performs well on small-tick stocks but fails on larger ticks like INTC. The benchmark reveals that while GenAI models can match many statistical properties, they still struggle with specific conditional distributions and long-horizon stability. The paper provides open-source code and generated data, establishing a new standard for evaluating synthetic financial data generation.

Research problem

The paper addresses the absence of consensus on quantitative evaluation paradigms for generative models of high-frequency financial data, particularly limit order books. It highlights that current evaluation methods, such as cross-entropy loss or qualitative checks of stylized facts, are insufficient for assessing the realism and stability of autoregressive generative models in open-loop sampling regimes.

Core contributions

• Introduction of LOB-Bench, a comprehensive benchmark for evaluating the quality and realism of generative LOB data.
• Development of a flexible framework for multivariate statistical evaluation, including unconditional and conditional distributional metrics.
• Implementation of market impact metrics to assess how well generated data reproduces price response functions to specific order events.
• Demonstration that autoregressive state-space models outperform traditional parametric and GAN-based models in generating realistic LOB sequences.
• Identification of "model derailment" as a common failure mode in generative LOB models, quantified through error divergence over prediction horizons.

Method

• Unconditional distributional evaluation using L1 and Wasserstein-1 distances on histogram estimates.
• Conditional evaluation by binning conditioning variables and computing weighted average distributional errors.
• Market impact response functions calculated using time-averaged mid-price changes following specific order events.
• Adversarial measurement using a Conv1D + Attention discriminator to find worst-case distributional differences.
• Error divergence analysis by comparing distributions conditional on the forecasting horizon step.
• Downstream task evaluation via MLP training for mid-price classification.

Results & findings

• LOBS5 achieves the lowest overall distributional error across L1 and Wasserstein-1 metrics compared to baselines.
• The baseline parametric model fails to reproduce market impact curves accurately, with high L1 distance.
• RWKV models exhibit rapid error accumulation, leading to significant distributional drift over longer sequences.
• The Coletta model performs well on GOOG but fails on INTC due to architectural limitations for large-tick stocks.
• Discriminator scores indicate that generated data is distinguishable from real data, setting a high bar for future models.
• Synthetic data from LOBS5 improves mid-price prediction F1-scores compared to using only real data.

Data & experiments

• High-frequency LOB data for Alphabet (GOOG) and Intel (INTC) stocks in LOBSTER format.
• Evaluation of four generative models: LOBS5 (state-space), Coletta (C-GAN), RWKV 4/6, and a parametric baseline.
• Metrics include L1 norm, Wasserstein-1 distance, discriminator ROC scores, and market impact response function errors.
• Conditional evaluation of statistics such as spread conditioned on time of day and volatility.
• Downstream task evaluation using mid-price prediction classifiers trained on real vs. synthetic data.

Limitations & caveats

• The benchmark is currently focused on single-asset LOB data; multi-asset extensions are not covered.
• The discriminator performance may vary across different stocks and time periods.
• The LOBS5 model's performance on INTC is slightly worse than on GOOG, indicating tick-size sensitivity.
• The evaluation relies on specific scoring functions; other relevant metrics might exist.
• The study does not explore the use of generated data for direct trading strategy optimization.

Curator notes

Discovery status: institutional_trading_round3_promoted. Source links: ABIDES: Towards High-Fidelity Multi-Agent Market Simulation; JAX-LOB: A GPU-Accelerated limit order book simulator to unlock large scale reinforcement learning for trading