Multimodal image fusion traditionally generates a single fused image from paired observations at a specific viewpoint, limiting comprehensive scene perception and adaptability to diverse perceptual objectives. To overcome these limitations, we present LIFT, a novel framework for our proposed language-controlled 3D image fusion (L3D-IF) paradigm that extends multimodal image fusion from 2D images to free-viewpoint 3D scenes, enabling language-guided fusion tailored to diverse perceptual objectives. First, LIFT formulates Semantic Modality Binding (SMB), which interprets language instructions to identify semantic targets and their associated modality preferences and grounds these targets in corresponding regions of a shared Gaussian scene, establishing persistent semantic-to-spatial correspondences for consistent fusion control across viewpoints. Second, Language-Conditioned Gaussian Appearance (LCGA) is developed to achieve scene-wide multimodal appearance modeling by combining adaptive multimodal blending with language- and viewpoint-conditioned appearance refinement, enabling spatially adaptive realization of diverse fusion appearances while preserving source fidelity at pure-modality endpoints. Finally, we construct a Multi-View Language-Supervision (MVLS) dataset that associates language instructions and calibrated viewpoints with modulated fused images, providing effective supervision for learning instruction-dependent fusion from finite multi-view observations. Extensive experiments demonstrate that LIFT achieves coherent multimodal fusion across continuous viewpoints, accurate semantic target localization, and consistent regional fusion control under diverse language instructions.
翻译:暂无翻译