GPT-4 Vision had better watch out: discover the open source alternatives to LLaVA 1.5 that are coming!
LLaVA 1.5: An open source alternative to GPT-4 Vision
Generative artificial intelligence is rapidly evolving with the emergence of multimodal language models (LMMs) such as OpenAI’s GPT-4 Vision. These models revolutionize our interaction with AI systems by integrating text and images.
However, the closed and commercial nature of some of these technologies may limit their universal adoption. It is in this context that the open source community comes into play, propelling the LLaVA 1.5 model as a promising alternative to GPT-4 Vision.
The mechanics of LMM
LMMs operate using a multi-layer architecture. They combine a pre-trained model to encode visual elements, a large language model (LLM) to interpret and respond to user instructions, and a multimodal connector to link vision and language.
Their training takes place in two stages: an initial phase of alignment between vision and language, followed by fine adjustment to respond to visual requests. This process, although efficient, often requires significant computational resources and a rich and precise database.
The advantages of LLaVA 1.5
LLaVA 1.5 relies on the CLIP model for visual encoding and Vicuna for language. Unlike the original LLaVA model, which used the text versions of ChatGPT and GPT-4 for visual adjustment, LLaVA 1.5 goes further by connecting the language model and visual encoder via a multi-layer perceptron (MLP). This update enriched its learning database with visual Q&As, totaling approximately 600,000 examples. LLaVA 1.5 thus outperformed other open source LMMs on 11 of 12 multimodal benchmarks.
The future of open source LMMs
The online demo of LLaVA 1.5, accessible to everyone, shows promising results, even on a limited budget. However, one restriction remains: the use of data generated by ChatGPT limits its use to non-commercial purposes.
Despite this limitation, LLaVA 1.5 provides a path to the future of open source LMMs. Its cost-effectiveness, ability to generate scalable learning data, and efficiency in adjusting visual instructions make it a foreshadow of future innovations.
LLaVA 1.5 is just the beginning of a series of advancements from the open source community. By anticipating more efficient and accessible models, we can envision a future where generative AI technology is accessible to everyone, revealing the limitless potential of artificial intelligence.