We present Cephalo, a series of multimodal vision large language models (V-LLMs) designed for materials science applications, integrating visual and linguistic data for enhanced understanding and interaction within human-AI and multi-agent AI frameworks. A key innovation of Cephalo is its advanced dataset generation method, which employs a sophisticated algorithm to accurately detect and separate images and their corresponding textual descriptions from PDF documents, such as scientific papers. The method includes a careful refinement of image-text pairs through integrated vision and language processing, ensuring high-quality, contextually relevant, and well reasoned training data. Cephalo is trained on integrated image and text data extracted from thousands of scientific papers and science-focused Wikipedia pages demonstrates can interpret complex visual scenes, generate precise language descriptions, and answer queries about images effectively. The combination of a vision encoder with an autoregressive transformer supports complex natural language understanding in an integrated model, which can be coupled with other generative methods to create an image-to-text-to-image or image-to-text-to-3D pipeline. To explore the development of larger models from smaller ones, we report both mixture-of-expert methods and model merging. These hybrid approaches allow us to leverage the domain-specific expertise and general conversational capabilities to harness the strengths of multiple models. We examine the models in diverse use cases that incorporate biological materials, fracture and engineering analysis, protein biophysics, and bio-inspired design based on insect behavior. Generative applications include bio-inspired designs, including pollen-inspired architected materials, as well as the synthesis of bio-inspired material microstructures from a photograph of a solar eclipse.

翻译：本文提出Cephalo系列多模态视觉大语言模型（V-LLMs），专为材料科学应用设计，通过整合视觉与语言数据以增强人机交互及多智能体AI框架中的理解与交互能力。Cephalo的核心创新在于其先进的数据集生成方法，该方法采用精密算法从PDF文档（如科研论文）中精确检测并分离图像及其对应文本描述。该技术通过集成视觉与语言处理对图文对进行精细化提炼，确保生成高质量、上下文相关且逻辑严谨的训练数据。基于从数千篇科学论文及科学类维基百科页面提取的图文融合数据训练的Cephalo模型，能够解析复杂视觉场景、生成精准语言描述，并有效回答图像相关查询。视觉编码器与自回归Transformer的结合在集成模型中实现了复杂的自然语言理解，该模型可与其他生成方法结合，构建图像-文本-图像或图像-文本-3D的生成流程。为探索从小模型扩展至大模型的路径，我们同时报告了专家混合方法与模型融合策略。这些混合方法使我们能够整合领域专业知识与通用对话能力，充分发挥多模型优势。我们在多种应用场景中验证模型性能，涉及生物材料、断裂与工程分析、蛋白质生物物理学，以及基于昆虫行为的仿生设计。生成式应用涵盖仿生设计领域，包括花粉启发结构材料，以及从日食照片合成仿生材料微结构。