FLAG-Trader is a novel framework that fuses large language models (LLMs) with gradient-based reinforcement learning (RL) for financial trading. The key innovation is using a partially fine-tuned LLM as the policy network within an actor-critic RL architecture. The LLM is split into frozen base layers (retaining pre-trained knowledge) and trainable top layers (adapted to finance via parameter-efficient fine-tuning). Market states are converted into structured text prompts, enabling the LLM to leverage its language reasoning for decision-making. The policy network outputs trading actions (buy, hold, sell) while a shared value network estimates state values. Both networks are optimized jointly using Proximal Policy Optimization (PPO), with gradients flowing into the trainable LLM layers to align representations with trading rewards.

Extensive experiments show that FLAG-Trader significantly outperforms traditional buy-and-hold strategies and existing LLM-agentic baselines (e.g., FinMem, TradingGPT) in terms of cumulative return and Sharpe ratio. Remarkably, a small 135M-parameter LLaMA model fine-tuned with FLAG-Trader surpasses much larger proprietary models like GPT-4, highlighting the effectiveness of RL-driven fine-tuning. Additionally, the RL fine-tuning improves performance on other financial tasks (e.g., sentiment analysis), demonstrating transferability. The framework balances computational efficiency and domain adaptation by updating only a subset of LLM parameters, making it practical for real-time trading.

Overall, FLAG-Trader provides a unified solution to integrate LLMs' reasoning capabilities with RL's reward-driven optimization, addressing key challenges in financial sequential decision-making such as multimodal data integration, non-stationarity, and manual feature engineering. The results underscore the potential of combining LLMs with RL for robust and efficient financial trading agents.