We present Cephalo, a series of multimodal vision large language models (V-LLMs) designed for materials science applications, integrating visual and linguistic data for enhanced understanding and interaction within human-AI and multi-agent AI frameworks. A key innovation of Cephalo is its advanced dataset generation method, which employs a sophisticated algorithm to accurately detect and separate images and their corresponding textual descriptions from PDF documents, such as scientific papers. The method includes a careful refinement of image-text pairs through integrated vision and language processing, ensuring high-quality, contextually relevant, and well reasoned training data. Cephalo is trained on integrated image and text data extracted from thousands of scientific papers and science-focused Wikipedia pages demonstrates can interpret complex visual scenes, generate precise language descriptions, and answer queries about images effectively. The combination of a vision encoder with an autoregressive transformer supports complex natural language understanding in an integrated model, which can be coupled with other generative methods to create an image-to-text-to-image or image-to-text-to-3D pipeline. To explore the development of larger models from smaller ones, we merge sets of layers that originate from different pre-trained source models. This hybrid approach allows us to leverage the domain-specific expertise and general conversational capabilities to harness the strengths of multiple models. We examine the models in diverse use cases that incorporate biological materials, fracture and engineering analysis, protein biophysics, and bio-inspired design based on insect behavior. Generative applications include bio-inspired designs, including pollen-inspired architected materials, as well as the synthesis of bio-inspired material microstructures from a photograph of a solar eclipse.

翻译：我们提出了头足类（Cephalo）系列多模态视觉大语言模型（V-LLMs），专为材料科学应用设计，通过整合视觉与语言数据，增强人机交互与多智能体AI框架下的理解与交互能力。该模型的核心创新在于其先进的数据集生成方法，该方法采用精密算法从PDF文档（如科研论文）中精确检测并分离图像及其对应的文本描述。此过程通过集成视觉与语言处理对图文对进行细致优化，确保获得高质量、上下文相关且逻辑清晰的训练数据。基于从数千篇科学论文及科学类维基百科页面提取的图文整合数据训练，头足类模型能够解析复杂视觉场景、生成精准语言描述，并有效回答关于图像的查询。视觉编码器与自回归Transformer的结合，在统一模型中实现了复杂的自然语言理解能力，该模型可与其他生成方法结合，构建图像-文本-图像或图像-文本-3D的生成流程。为探索从小模型扩展至大模型的路径，我们融合了来自不同预训练源模型的层级集合。这种混合策略使我们能够综合利用领域专业知识与通用对话能力，发挥多模型优势。我们在多种应用场景中评估模型性能，涉及生物材料、断裂与工程分析、蛋白质生物物理学，以及基于昆虫行为的仿生设计。生成式应用包括仿生设计（如受花粉启发的结构材料），以及从日食照片合成仿生材料微结构。