Using huggingface transformers library, I see different outputs when generating text with model.generate with and without use_cache argument. Is this intended and how can I combat this? The scores when I use cache (from the second token generated and onwards) are different. AFAIK use_cache is an op…

Similar case? [image] Transformer KV-Cache Produces Worse Output Than Normal Generation – Why? Beginners Hi everyone, I implemented a transformer model from scratch based on Andrej Karpathy’s video. After that, I added some improvements, such as parallelized multi-head at…

This case talks about a problem with his custom implementation. I am facing a problem with the official model.generate function in huggingface transformers.

I found a cause. Possible Bug with KV Caching in Llama (original) model opened 07:50PM - 09 Aug 23 UTC (UTC) closed 08:07AM - 09 Jan 24 UTC (UTC) [image] maximkha …

Model.generate use_cache=True generates different results than use_cache=False

Intermediate

John6666 March 4, 2025, 1:24pm 2

Similar case?

Topic		Replies	Views
Outputs change if re-using KVCache (past_key_values) for model.forward and generation 🤗Transformers	5	553	January 22, 2025
Transformer KV-Cache Produces Worse Output Than Normal Generation – Why? Beginners	1	543	March 3, 2025
What is the purpose of 'use_cache' in decoder? 🤗Transformers	5	25109	July 4, 2023
IndexError: index -1 is out of bounds for dimension 0 with size 0 🤗Transformers	3	245	November 7, 2025
What does the `use_cache` in `generate` actually do? 🤗Transformers	1	2670	May 9, 2024

Model.generate use_cache=True generates different results than use_cache=False

Related topics