A Scoping Review of ChatGPT Research in Accounting and Finance

In one sentence

Summary

This paper addresses the rapid proliferation of Large Language Model (LLM) research in accounting and finance by conducting a scoping review of 264 publications and working papers from January 2022 to March 2024. The authors aim to map the current state of the art, identify knowledge gaps, and provide technical guidance for researchers. Unlike systematic reviews that assess efficacy, this study focuses on the breadth of emerging literature to inform future research agendas, utilizing an input-process-output framework inspired by technology adoption theories. The review categorizes studies based on their motivation for adoption, the specific LLM capabilities leveraged, and the maturity of adoption implications, ranging from conceptual discussions to value realization. The methodology involves a structured search across SSRN and published journals, filtering for relevance to accounting and finance. The authors organize the literature into three primary themes: applications of LLMs in specific fields (audit, reporting, asset pricing), the use of LLMs as research tools for classification and summarization, and the implications of LLM adoption for professionals and organizations. The analysis draws on the Gartner Hype Cycle to contextualize the adoption stage, noting a shift from early conceptual papers to potential applications. The review also includes a technical appendix detailing how to use LLM APIs, handle embeddings, and mitigate risks like hallucinations, offering practical guidance for empirical research design. Key findings indicate that research is heavily concentrated in audit, financial reporting, asset pricing, and corporate finance. LLMs are primarily used for text generation, sentiment analysis, and classification, often outperforming traditional methods in efficiency. The paper highlights that ChatGPT-4 can pass professional exams like the CPA, suggesting significant shifts in labor markets and education. However, the review notes limitations in current studies, including a lack of large-scale empirical evidence on financial payoffs and concerns about data privacy and hallucination. The authors propose future research avenues, such as investigating the long-term impact on labor productivity and developing domain-specific fine-tuning strategies, emphasizing the need for rigorous evaluation of LLM outputs in high-stakes financial contexts.

Research problem

The rapid emergence of ChatGPT and LLMs has generated a surge in academic research within accounting and finance, but the literature remains fragmented and rapidly evolving. There is a need to synthesize this burgeoning body of work to understand the current state of research, identify common themes, and pinpoint gaps. The paper addresses the challenge of organizing this diverse literature to provide clarity on how LLMs are being applied, used as tools, and impacting the profession, while offering guidance for future scholarly inquiry.

Core contributions

• Provides a comprehensive scoping review of 264 papers on ChatGPT/LLMs in accounting and finance (2022-2024).
• Introduces an input-process-output framework to categorize LLM research based on motivation, capability usage, and adoption maturity.
• Identifies three dominant themes: domain applications, LLMs as research tools, and professional/organizational implications.
• Offers technical guidance for researchers on using LLM APIs, embeddings, and mitigating risks like hallucinations.
• Maps the evolution of research from conceptual discussions to potential applications, aligning with technology adoption cycles.
• Proposes specific future research avenues, including labor market impacts and domain-specific model development.

Method

• Conducts a scoping review following established protocols (Paré et al. 2015; Snyder 2019).
• Develops a framework combining technology adoption theory (O'Leary 2008) with an input-process-output model.
• Searches SSRN and published journals for papers related to ChatGPT and LLMs in accounting/finance.
• Categorizes studies by application domain, LLM capability used, and adoption maturity stage.
• Analyzes trends in research focus over time, noting the shift from conceptual to applied studies.
• Provides a technical appendix detailing API usage, embedding generation, and risk mitigation strategies.

Results & findings

• Research is heavily concentrated in four areas: audit, financial reporting, asset pricing, and corporate finance.
• LLMs are primarily leveraged for text generation, classification, and summarization tasks in existing studies.
• ChatGPT-4 demonstrates capability to pass professional exams like the CPA, indicating high proficiency in domain knowledge.
• Early evidence suggests LLM-augmented professionals may outperform non-users, pointing to workflow shifts.
• The literature shows a transition from conceptual papers to studies exploring potential applications and value realization.
• Significant gaps remain in large-scale empirical studies measuring financial payoffs and long-term adoption impacts.
• Hallucination and data privacy are identified as key risks that current studies often overlook or address superficially.

Data & experiments

• Scoping review of 264 papers from SSRN and published journals in Accounting, Finance, and Economics networks.
• Timeframe covers January 2022 to March 2024, capturing the rapid post-launch surge in LLM research.
• Analysis categorizes studies by application domain (audit, reporting, asset pricing, etc.) and LLM capability (classification, summarization, etc.).
• Uses the Gartner Hype Cycle as a theoretical lens to assess the stage of technology adoption in the literature.
• Includes a technical appendix evaluating practical aspects of LLM usage, such as API integration and prompt engineering.

Limitations & caveats

• As a scoping review, it does not provide rigorous meta-analysis or quantitative synthesis of effect sizes.
• Relies on working papers (SSRN) which may not have undergone peer review, potentially including preliminary or unverified findings.
• The rapid pace of LLM development means some specific model capabilities discussed may become outdated quickly.
• Limited discussion on the long-term financial performance impacts of LLM adoption due to the nascent stage of research.
• Potential bias towards positive aspects of LLMs in early-stage studies, as noted by the authors.
• Does not provide a comprehensive evaluation of specific LLM models' performance on standardized financial benchmarks.

Curator notes

Discovery status: round3_promoted. Source links: Assessing Look-Ahead Bias in Stock Return Predictions Generated by GPT Sentiment Analysis; FinBERT : A Large Language Model for Extracting Information from Financial Text†; GPT-InvestAR: Enhancing Stock Investment Strategies through Annual Report Analysis with Large Language Models; Instruct-FinGPT: Financial Sentiment Analysis by Instruction Tuning of General-Purpose Large Language Models