Generative AI and large language models hold great promise in enhancing programming education by generating individualized feedback and hints for learners. Recent works have primarily focused on improving the quality of generated feedback to achieve human tutors' quality. While quality is an important performance criterion, it is not the only criterion to optimize for real-world educational deployments. In this paper, we benchmark language models for programming feedback generation across several performance criteria, including quality, cost, time, and data privacy. The key idea is to leverage recent advances in the new paradigm of in-browser inference that allow running these models directly in the browser, thereby providing direct benefits across cost and data privacy. To boost the feedback quality of small models compatible with in-browser inference engines, we develop a fine-tuning pipeline based on GPT-4 generated synthetic data. We showcase the efficacy of fine-tuned Llama3-8B and Phi3-3.8B 4-bit quantized models using WebLLM's in-browser inference engine on three different Python programming datasets. We will release the full implementation along with a web app and datasets to facilitate further research on in-browser language models.

翻译：生成式AI与大语言模型通过为学习者提供个性化反馈与提示，在提升编程教育方面展现出巨大潜力。现有研究主要致力于提升生成反馈的质量以达到人类导师的水平。尽管质量是重要的性能指标，但在实际教育部署中并非唯一需要优化的标准。本文针对编程反馈生成任务，从质量、成本、时间及数据隐私等多个性能维度对语言模型进行基准评估。核心思路是利用浏览器内推理这一新范式的近期进展，使模型可直接在浏览器中运行，从而在成本与数据隐私方面获得直接优势。为提升兼容浏览器内推理引擎的小型模型的反馈质量，我们基于GPT-4生成的合成数据构建了微调流水线。通过使用WebLLM的浏览器内推理引擎，我们展示了经过微调的Llama3-8B与Phi3-3.8B（4比特量化）模型在三个不同Python编程数据集上的有效性。我们将公开完整实现代码、Web应用及数据集，以促进浏览器内语言模型的进一步研究。