Exploring the Necessity of Annotation in Multi-Modal LLM Fine-Tuning for Enhanced Image Comprehension

[John6666]

There seems to be a post on a similar subject. Let’s join them over there.

Multi modal LLM fine tuning (image annotation)

Is Annotation Necessary for the Multi Model LLM to Learn the Details in the Image? or beneficial ? For example, when I provide an image and ask, there is a speed sign on the left side, there are children eating ice cream on the right side, the stork is flying above, the weather is sunny, the ground is asphalt and pavement, etc. Do I need to anotate these for the model to learn them? I have seen some fine tuning examples in which they only use image and text without annotation.