Hi @ydshieh , I am trying to fine-tune Kosmos-2 model on DocLayNet dataset. It has image, bounding box and its label. I want to create token classifier. I have converted dataset into Kosmos-2 required string. I am getting below error: ValueError: The model did not return a loss from the inputs, on…

[image] Mit1208: default_data_collator well, I don’t add default_data_collator: it’s on your notebook already (I don’t know if it’s necessary however :grin: )

I will add the below code to handle padding. Let me know if it’s a wrong way. labels = inputs['input_ids'].clone() labels[inputs['attention_mask'] == 0] = -100 inputs['labels'] = labels

I think the padding token id is 1 instead of 0. You can see that in Kosmos2TextConfig but as well as by checking the example you have in the notebook. Other than this, I think it it!

I was using attention_mask so it was working with 0 but for input_ids it’s 1. I checked config file, padding token id is 1. new code would look like this: labels = inputs['input_ids'].clone() labels[inputs['input_ids'] == 1] = -100 inputs['labels'] = labels

@Mit1208 Did you successfully enable FP16 at your training notebook?

@Mit1208 I just changed add a line before 1152 at modeling_kosmos2.py to temporarily enable fp16 inputs_embeds = inputs_embeds.to(image_embeds.dtype)

Hello everyone @Mit1208 and @ydshieh I am still eager to see if we can adapt Kosmos2 for my task :smiley: Did you get any working examples yet? I see the Colab Notebook you shared, but I don’t know if it’s updated. Any news? Thanks, CDH

hi @cdh I think I have everything and running fine thanks to @ydshieh . I couldn’t train model just because of GPU resource but I can share colab which has everything. I just need to adjust few things. I will try to share in few hours.

Hi @cdh Here is my final code which shows kosmos-2 fine-tuning. because of the GPU limitation I updated the layers information. You can remove the Config parameter while loading the model and you are good to go. My code: Finetuning code

Kosmos-2 Fine tuning

🤗Transformers

Mit1208 March 25, 2024, 5:58pm 25

@yuerlong fp16=True in training argument was giving me an error so I removed that argument.

FYI - Issue on Kosmos-2 model training on new dataset

Topic		Replies	Views
Issue on Kosmos-2 model training on new dataset 🤗Transformers	3	480	February 25, 2024
Issue with KOSMOS-2 encoding and decoding 🤗Tokenizers	11	560	January 26, 2024
Finetuning GPT2 with user defined loss Beginners	56	16474	July 23, 2023
Need help in dealing with out of bounds Beginners	16	260	December 23, 2024
The model did not return a loss from the inputs, only the following keys: logits. For reference, the inputs it received are input_values Intermediate	24	42288	January 23, 2026

Kosmos-2 Fine tuning

Related topics