Toward Reliable Evaluation of LLM-Based Financial Multi-Agent Systems: Taxonomy, Coordination Primacy, and Cost Awareness
Toward Reliable Evaluation of LLM-Based Financial Multi-Agent Systems: Taxonomy, Coordination Primacy, and Cost Awareness
This survey addresses the lack of reliable evaluation frameworks for LLM-based financial multi-agent systems (MAS). The authors introduce a four-dimensional taxonomy covering architecture pattern, coordination mechanism, memory architecture, and tool integration, applied to 12 multi-agent systems and two single-agent baselines. They identify five systematic evaluation failures—look-ahead bias, survivorship bias, backtesting overfitting, transaction cost neglect, and regime-shift blindness—that can reverse the sign of reported returns, making cross-system comparisons unreliable. Building on structural patterns that survive these failures, they formulate the Coordination Primacy Hypothesis (CPH): inter-agent coordination protocol is the most consequential design factor for trading decision quality, often more important than model scaling. The CPH is presented as a falsifiable hypothesis supported by tiered evidence (live-market benchmarks, ablation studies, theoretical arguments) but not yet empirically validated due to evaluation infrastructure gaps. To operationalize the CPH, they introduce the Coordination Breakeven Spread (CBS) metric, which determines whether coordination adds genuine value net of transaction costs by comparing coordination-driven signal improvement to the bid-ask spread. The paper concludes with minimum evaluation standards (contamination control, point-in-time universe, rolling-window reporting, net-of-cost returns, regime coverage) as prerequisites for validating the CPH, and identifies three future directions: controlled CPH validation, small language model specialist architectures, and systemic risk from correlated AI trading.
Highlights
- 1Introduces a four-dimensional taxonomy (architecture, coordination, memory, tools) for LLM-based financial multi-agent systems, applied to 12 systems and 2 baselines.
- 2Formulates the Coordination Primacy Hypothesis (CPH): inter-agent coordination protocol is the most consequential design factor for trading decision quality, often more important than model scaling.
- 3Documents five pervasive evaluation failures (look-ahead bias, survivorship bias, backtesting overfitting, transaction cost neglect, regime-shift blindness) that can reverse reported returns.
- 4Proposes the Coordination Breakeven Spread (CBS) metric to determine whether multi-agent coordination adds net value after transaction costs.
- 5Provides minimum evaluation standards (contamination control, point-in-time universe, rolling windows, net-of-cost returns, regime coverage) as prerequisites for validating the CPH.
Methods
- MTaxonomic classification across four dimensions: architecture pattern, coordination mechanism, memory architecture, and tool integration.
- MCritical analysis of evaluation methodologies, identifying five systematic failures and their impact on reported performance.
- MFormulation of the Coordination Primacy Hypothesis (CPH) as a falsifiable claim supported by tiered structural evidence.
- MDevelopment of the Coordination Breakeven Spread (CBS) metric to quantify the economic viability of coordination relative to transaction costs.
Results
- RNo surveyed system satisfies all five proposed evaluation standards; the best (FinCon, HedgeAgents, FinAgent, QuantAgents) meet only 2/5 criteria.
- RFinMem's reported 23% return on MSFT reversed to -22% under controlled re-evaluation with proper methodology.
- RSystems with explicit coordination mechanisms (e.g., HedgeAgents, QuantAgents) more often report extended evaluation horizons and live deployment, while pipeline designs rarely do.
- RAblation studies in FinCon and TradingAgents show coordination removal reduces Sharpe by 15–30%, while model substitution yields only 5–8% variance.
- RLive-market benchmarks suggest weaker models within sophisticated coordination structures outperform frontier models in linear pipelines.
Analyze Paper
Generate insights from "Toward Reliable Evaluation of LLM-Based Financial Multi-Agen...".