Task embedding, a meta-learning technique that captures task-specific information, has gained popularity, especially in areas such as multi-task learning, model editing, and interpretability. However, it faces challenges with the emergence of prompt-guided Large Language Models (LLMs) operating in a gradient-free manner. Existing task embedding methods rely on fine-tuned, task-specific language models, which hinders the adaptability of task embeddings across diverse models, especially prompt-based LLMs. To hardness the potential of task embeddings in the era of LLMs, we propose a framework for unified task embeddings (FUTE), harmonizing task embeddings from various models, including smaller language models and LLMs with varied prompts, within a single vector space. Such uniformity enables comparison and analysis of similarities amongst different models, broadening the scope and utility of existing task embedding methods in multi-model scenarios, while maintaining their performance comparable to architecture-specific methods.

翻译：任务嵌入作为一种捕捉任务特定信息的元学习技术，已在多任务学习、模型编辑和可解释性等领域广受欢迎。然而，随着以无梯度方式运行的提示引导大型语言模型（LLMs）的出现，该技术面临新的挑战。现有任务嵌入方法依赖于经过微调的任务特定语言模型，这阻碍了任务嵌入在不同模型（尤其是基于提示的LLMs）间的适应性。为充分发挥任务嵌入在LLM时代的潜力，我们提出了统一任务嵌入框架（FUTE），将来自不同模型（包括小型语言模型和采用多样化提示的LLMs）的任务嵌入协调至同一向量空间。这种统一性使得能够比较和分析不同模型间的相似性，从而在多模型场景中拓宽现有任务嵌入方法的适用范围和实用性，同时保持其性能与特定架构方法相当。