PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance
PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance
This paper introduces PIXIU, a comprehensive framework addressing the lack of publicly available resources for financial artificial intelligence development. The authors make three key contributions: FinMA (the first financial large language model fine-tuned from LLaMA), a large-scale instruction dataset with 136K samples covering diverse financial tasks and document types, and a standardized evaluation benchmark comprising 5 tasks across 9 datasets. These resources fill a critical gap in open-source financial AI development.
The methodology involves constructing multi-task instruction data considering various financial tasks, document types, and data modalities, then fine-tuning LLaMA to create FinMA. The evaluation benchmark covers both financial NLP tasks and financial prediction tasks, enabling systematic assessment of financial LLMs. Through comparative analysis using this benchmark, the paper provides detailed insights into the performance characteristics of different LLMs on critical financial tasks. All components of the PIXIU framework are released as open-source resources to facilitate future research and development in financial AI.
Highlights
- 1First publicly available financial LLM (FinMA) fine-tuned from LLaMA
- 2First large-scale financial instruction dataset with 136K samples covering multiple tasks and modalities
- 3First comprehensive evaluation benchmark with 5 tasks and 9 datasets for financial AI
- 4Open-source release of model, data, benchmark, and results to advance financial AI research
- 5Detailed comparative analysis of financial LLMs revealing strengths and weaknesses
Methods
- MInstruction tuning of LLaMA model with financial domain data
- MConstruction of multi-task instruction dataset covering diverse financial tasks and document types
- MDevelopment of standardized evaluation benchmark with financial NLP and prediction tasks
- MComparative analysis of multiple LLMs using the proposed benchmark
Results
- RFinMA demonstrates capability to follow instructions for various financial tasks
- RBenchmark reveals significant performance variations among LLMs across different financial tasks
- ROpen-source resources enable reproducible research and development in financial AI
- RAnalysis identifies specific financial tasks where current LLMs show strengths and limitations
- RFramework provides foundation for future improvements in financial language models
Analyze Paper
Generate insights from "PIXIU: A Large Language Model, Instruction Data and Evaluati...".