bitsandbytes is an open-source Python library for PyTorch, maintained by the bitsandbytes Foundation. It does k-bit quantization: it stores a model's weights with fewer bits, for example 8 or 4 bits instead of 16 or 32, so that a large language model needs much less memory.
It has three main features. LLM.int8() loads a model in 8-bit form for inference with about half the memory; it quantizes most values to 8 bits and handles the rare large values (outliers) with 16-bit matrix multiplication. QLoRA loads a model in 4-bit form and adds a small set of trainable LoRA (low-rank adaptation) weights, so that the model can be fine-tuned with little memory. 8-bit optimizers keep the optimizer state, the extra numbers an optimizer stores for each weight during training, in 8 bits. The library works with Hugging Face Transformers, Diffusers and PEFT, and runs on NVIDIA, AMD and Intel GPUs and on CPUs.
It is free software under the MIT license, developed in the open on GitHub and supported by sponsors.