Large Language Models (LLMs) have recently been leveraged for asset pricing tasks and stock trading applications, enabling AI agents to generate investment decisions from unstructured financial data. However, most evaluations of LLM timing-based investing strategies are conducted on narrow timeframes and limited stock universes, overstating effectiveness due to survivorship and data-snooping biases. We critically assess their generalizability and robustness by proposing FINSABER, a backtesting framework evaluating timing-based strategies across longer periods and a larger universe of symbols. Systematic backtests over two decades and 100+ symbols reveal that previously reported LLM advantages deteriorate significantly under broader cross-section and over a longer-term evaluation. Our market regime analysis further demonstrates that LLM strategies are overly conservative in bull markets, underperforming passive benchmarks, and overly aggressive in bear markets, incurring heavy losses. These findings highlight the need to develop LLM strategies that are able to prioritise trend detection and regime-aware risk controls over mere scaling of framework complexity.

翻译：近年来，大型语言模型（LLMs）已被应用于资产定价任务和股票交易应用，使AI智能体能够从非结构化金融数据中生成投资决策。然而，大多数基于LLM择时投资策略的评估仅在狭窄的时间范围和有限的股票池中进行，由于幸存者偏差和数据窥探偏差而夸大了其有效性。我们通过提出FINSABER（一个在更长周期和更大股票代码池中评估择时策略的回测框架）来严格评估其泛化能力和稳健性。在跨越二十年、涵盖100多个股票代码的系统性回测中，我们发现先前报道的LLM优势在更广泛的横截面和更长期的评估下显著减弱。我们的市场状态分析进一步表明，LLM策略在牛市中过于保守，表现逊于被动基准；在熊市中则过于激进，导致重大亏损。这些发现强调了开发LLM策略的必要性，这些策略应优先关注趋势检测和状态感知的风险控制，而非仅仅增加框架复杂性。