The paper introduces FinBen, a comprehensive open-source benchmark designed to evaluate large language models (LLMs) in financial applications. Addressing the lack of standardized evaluation frameworks in finance, FinBen encompasses 36 datasets spanning 24 tasks across seven critical aspects: information extraction, textual analysis, question answering, text generation, risk management, forecasting, and decision-making. Key innovations include the first evaluation of stock trading, novel agent and Retrieval-Augmented Generation (RAG) assessments, and three new open-source datasets for text summarization, question answering, and stock trading.

The evaluation of 15 representative LLMs, including GPT-4, ChatGPT, and Gemini, reveals significant insights into their financial capabilities. While LLMs demonstrate strong performance in information extraction and textual analysis, they struggle with advanced reasoning and complex tasks such as text generation and forecasting. GPT-4 excels in information extraction and stock trading, whereas Gemini shows superior performance in text generation and forecasting. Instruction-tuned models improve textual analysis but offer limited advantages for complex question answering. FinBen's impact is further demonstrated through its use in hosting the first financial LLMs shared task at IJCAI-2024, where 12 teams developed novel solutions that outperformed GPT-4, highlighting the benchmark's potential to foster innovation in financial LLMs. All datasets, results, and code are publicly available to support ongoing research.