Malware open-set recognition (MOSR) aims at jointly classifying malware samples from known families and detect the ones from novel unknown families, respectively. Existing works mostly rely on a well-trained classifier considering the predicted probabilities of each known family with a threshold-based detection to achieve the MOSR. However, our observation reveals that the feature distributions of malware samples are extremely similar to each other even between known and unknown families. Thus the obtained classifier may produce overly high probabilities of testing unknown samples toward known families and degrade the model performance. In this paper, we propose the Multi-modal Dual-Embedding Networks, dubbed MDENet, to take advantage of comprehensive malware features (i.e., malware images and malware sentences) from different modalities to enhance the diversity of malware feature space, which is more representative and discriminative for down-stream recognition. Last, to further guarantee the open-set recognition, we dually embed the fused multi-modal representation into one primary space and an associated sub-space, i.e., discriminative and exclusive spaces, with contrastive sampling and rho-bounded enclosing sphere regularizations, which resort to classification and detection, respectively. Moreover, we also enrich our previously proposed large-scaled malware dataset MAL-100 with multi-modal characteristics and contribute an improved version dubbed MAL-100+. Experimental results on the widely used malware dataset Mailing and the proposed MAL-100+ demonstrate the effectiveness of our method.

翻译：恶意软件开放集识别（MOSR）旨在对已知家族的恶意软件样本进行分类，同时检测来自未知新颖家族的样本。现有方法多依赖训练良好的分类器，通过基于阈值的检测机制结合各已知家族的预测概率来实现MOSR。然而，我们的观察发现，恶意软件样本的特征分布极度相似，即便是已知家族与未知家族之间亦是如此。由此获得的分类器可能对未知测试样本产生过高的已知家族概率，进而降低模型性能。本文提出多模态双嵌入网络（MDENet），利用不同模态的全面恶意软件特征（即恶意软件图像与恶意软件语句）增强恶意软件特征空间的多样性，使其对下游识别任务更具代表性与区分性。最后，为保障开放集识别性能，我们将融合后的多模态表示双嵌入至主空间与关联子空间（即判别性空间与排他性空间），通过对比采样与ρ边界包围球正则化分别实现分类与检测功能。此外，我们进一步丰富了此前提出的大规模恶意软件数据集MAL-100的多模态特性，贡献了改进版本MAL-100+。在广泛使用的恶意软件数据集Mailing及所提MAL-100+上的实验结果表明了本方法的有效性。