Explicitly disable bf16 for some layers

[John6666]

Or similar to this issue…?

Why do you need to re-upcast the norm layers of HF falcon to fb32?

I saw these lines: bnb_4bit_compute_dtype=torch.float16, ... optim = "paged_adamw_32bit" ... for name, module in trainer.model.named_modules(): if "norm" in name: module = module.to(torch.float32) in the falcon tutorial. These are confusing me. Based on the original QLoRA tutorial, they use 4 bit model + during training they use 16 brain float (not normal float nor float 32). See equation 5: YBF16 = XBF16doubleDequant(c_FP32_1, c_k-bit_2, WNF4) + XBF16L_BF16_1 L_BF16_2 doubleDequa…