The BERT models I have found in the :hugs: Model’s Hub handle a maximum input length of 512. Using sequences longer than 512 seems to require training the models from scratch, which is time consuming and computationally expensive. However, the only limitation to input sequences longer than 512 in a …

In my experience, LongFormer and BigBird require a lot of GPU memory. I tried using these on a 14GB GPU, but I was limited to batch_size=1, which took for ever to train and yielded rather poor results.

Interesting, was just thinking that if a bert model is passed in longer sequences of tokens. But the bert model was initially pretrained with a limited (512) tokens. Wouldn’t the weights be “confused” by the longer sequences. hence perform badly?

Perhaps, but I assume the “confusion” will not be significant, given that the only element in BERT’s architecture that depends on the sequence length is the position embeddings. My empirical test on an IMDB text classification task suggests that using longer sequences does not degrade the performanc…

do you know if the max seq length is 512 character token or word token?

HI @moma1820 , I saw you posted a similar question in another thread, so I replied there. See the link below: [image] Is 512 token in bert, token or character level? Beginners BERT uses a subword tokenizer (WordPiece), so the maximum length corresponds to 512 subword token…

Fine-tuning BERT with sequences longer than 512 tokens

Models

arteagac April 3, 2022, 8:33pm 7

HI @moma1820, I saw you posted a similar question in another thread, so I replied there. See the link below:

Topic		Replies	Views
Is there a pre-trained BERT model with the sequence length 2048? 🤗Transformers	2	2176	November 5, 2020
Token Classification Models on (Very) Long Text Models	8	11664	March 9, 2023
Modeling long sequences Models	0	486	June 9, 2022
Flan-T5 - Finetuning to a Longer Sequence Length (512 -> 2048 tokens): Will it work? Beginners	3	4434	January 9, 2024
Passing Inputs Longer Than 512 Tokens After Pretraining a T5 Model: Is It Safe? 🤗Transformers	3	218	November 20, 2025

Fine-tuning BERT with sequences longer than 512 tokens

Related topics