The rapid proliferation of generative AI has raised questions about the competitiveness of lower-parameter, locally tunable, open-weight models relative to high-parameter, API-guarded, closed-weight models in terms of performance, domain adaptation, cost, and generalization. Centering under-resourced yet risk-intolerant settings in government, research, and healthcare, we see for-profit closed-weight models as incompatible with requirements for transparency, privacy, adaptability, and standards of evidence. Yet the performance penalty in using open-weight models, especially in low-data and low-resource settings, is unclear. We assess the feasibility of using smaller, open-weight models to replace GPT-4-Turbo in zero-shot, few-shot, and fine-tuned regimes, assuming access to only a single, low-cost GPU. We assess value-sensitive issues around bias, privacy, and abstention on three additional tasks relevant to those topics. We find that with relatively low effort, very low absolute monetary cost, and relatively little data for fine-tuning, small open-weight models can achieve competitive performance in domain-adapted tasks without sacrificing generality. We then run experiments considering practical issues in bias, privacy, and hallucination risk, finding that open models offer several benefits over closed models. We intend this work as a case study in understanding the opportunity cost of reproducibility and transparency over for-profit state-of-the-art zero shot performance, finding this cost to be marginal under realistic settings.

翻译：生成式人工智能的快速扩散引发了一个问题：在性能、领域适应、成本和泛化能力方面，参数较少、可本地调优的开放权重模型相对于参数庞大、API受保护、权重封闭的模型是否具有竞争力。聚焦于政府、研究和医疗保健领域中资源不足但风险承受能力低的环境，我们认为营利性的封闭权重模型与透明度、隐私性、适应性和证据标准的要求不相容。然而，使用开放权重模型所带来的性能损失，尤其是在低数据和低资源环境下，尚不明确。我们评估了使用较小的开放权重模型在零样本、少样本和微调机制下替代GPT-4-Turbo的可行性，假设仅能访问单个低成本GPU。我们还针对与偏见、隐私和弃权相关的三个额外任务，评估了涉及价值敏感性的问题。我们发现，以相对较低的努力、极低的绝对货币成本以及相对较少的微调数据，小型开放权重模型可以在不牺牲通用性的前提下，在领域适应任务中实现具有竞争力的性能。随后，我们进行了考虑偏见、隐私和幻觉风险等实际问题的实验，发现开放模型相比封闭模型具有若干优势。本研究旨在作为一个案例研究，以理解可复现性和透明度相对于营利性最先进零样本性能的机会成本，并发现该成本在实际环境下是微乎其微的。