GIT-Mol: A multi-modal large language model for molecular science with graph, image, and text

What is it about?

Large language models have made significant strides in natural language processing, enabling innovative applications in molecular science by processing textual representations of molecules. However, most existing language models cannot capture the rich information with complex molecular structures or images. In this paper, we introduce GIT-Mol, a multi-modal large language model that integrates the Graph, Image, and Text information. To facilitate the integration of multi-modal molecular data, we propose GIT-Former, a novel architecture that is capable of aligning all modalities into a unified latent space. We achieve a 5%–10% accuracy increase in properties prediction and a 20.2% boost in molecule generation validity compared to the baselines. With the any-to-language molecular translation strategy, our model has the potential to perform more downstream tasks, such as compound name recognition and chemical reaction prediction.

Featured Image

Photo by National Cancer Institute on Unsplash

Photo by National Cancer Institute on Unsplash

Why is it important?

Molecular science covers a broad spectrum of fields that study the structures, properties, and interactions of molecules. It is a interdisciplinary field that draws on chemistry, physics, biology, and computer science. Molecular science is pivotal in drug discovery applications, such as target identification and validation, structure-based drug design, and side effect prediction. However, most existing methods of discovering new molecules or tweaking existing ones can be time-consuming, expensive, and prone to failure [1]. More recently, computational methods have shown significant advantages in molecule generation and tweaking [2]. These techniques enable rapid identification and optimization of potential drug candidates. However, these computational methods are limited by substantial computational demands.

Perspectives