Task embedding, a meta-learning technique that captures task-specific information, has become prevalent, especially in areas such as multi-task learning, model editing, and interpretability. However, it faces challenges with the emergence of prompt-guided Large Language Models (LLMs) operating in a gradientfree manner. Existing task embedding methods rely on fine-tuned, task-specific language models, which hinders the adaptability of task embeddings across diverse models, especially prompt-based LLMs. To unleash the power of task embedding in the era of LLMs, we propose a framework for unified task embeddings (FUTE), harmonizing task embeddings from various models, including smaller language models and LLMs with varied prompts, within a single vector space. Such uniformity enables the comparison and analysis of similarities amongst different models, extending the scope and utility of existing task embedding methods in addressing multi-model scenarios, whilst maintaining their performance to be comparable to architecture-specific methods.

翻译：任务嵌入作为一种捕获任务特定信息的元学习技术，已在多任务学习、模型编辑和可解释性等领域广泛应用。然而，随着基于提示驱动且在无梯度模式下运行的大型语言模型（LLMs）的兴起，这一方法面临挑战。现有任务嵌入方法依赖经过微调、针对特定任务的语言模型，这限制了任务嵌入在不同模型（尤其是基于提示的LLMs）间的适应性。为释放LLMs时代任务嵌入的潜力，我们提出了一种统一任务嵌入框架（FUTE），旨在将来自各类模型（包括较小语言模型及使用不同提示的LLMs）的任务嵌入协调至同一向量空间中。这种统一性使得不同模型间的相似性比较与分析成为可能，从而拓展了现有任务嵌入方法在多模型场景下的适用范围与实用性，同时保持其性能与专为特定架构设计的方法相当。