LLaVA (Large Language and Vision Assistant) is an open vision-language model, a model that takes an image together with a text instruction and answers in text. Its authors are Haotian Liu, Chunyuan Li, Qingyang Wu and Yong Jae Lee of the University of Wisconsin-Madison, Microsoft Research and Columbia University; the first paper was presented at NeurIPS 2023.
LLaVA connects a pre-trained CLIP ViT-L/14 visual encoder (a network that turns an image into numeric features) to the language model Vicuna through a simple projection matrix. Training has two stages: first only the projection is trained, so that image features line up with the language model; then the projection and the language model are fine-tuned on 158,000 image-and-instruction examples that GPT-4 generated. Later versions, LLaVA-1.5 and LLaVA-NeXT, improved accuracy, processed more pixels per image and added larger language models such as Llama 3 and Qwen.
The code is open source under the Apache 2.0 license, and the model weights and training data are free to download. The authors state that data, code and checkpoints are licensed for research use only.