Update README.md
Browse files
README.md
CHANGED
|
@@ -16,7 +16,7 @@ base_model:
|
|
| 16 |
- mistralai/Mistral-Small-24B-Instruct-2501
|
| 17 |
pipeline_tag: text-classification
|
| 18 |
---
|
| 19 |
-
# 🔥 Quantized Model: Mistral-Small-24B-Instruct-
|
| 20 |
|
| 21 |
This is a 4-bit quantized version of [mistralai/Mistral-Small-24B-Instruct-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501) model, quantized by [ConfidentialMind.com](https://www.confidentialmind.com) 🤖✨
|
| 22 |
It leverages the open-source GPTQModel quantization to achieve 4-bit precision with a group size of 128 resulting in a
|
|
@@ -26,6 +26,7 @@ faster model with minimal performance degradation.
|
|
| 26 |
Ran on a single NVIDIA A100 GPU with 80GB of VRAM.
|
| 27 |
|
| 28 |
*Note* `batch_size` is set quite high as the model is small, you may need to adjust this to your GPU VRAM.
|
|
|
|
| 29 |
|
| 30 |
## Model Details
|
| 31 |
- **Original Model:** [mistralai/Mistral-Small-24B-Instruct-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501)
|
|
|
|
| 16 |
- mistralai/Mistral-Small-24B-Instruct-2501
|
| 17 |
pipeline_tag: text-classification
|
| 18 |
---
|
| 19 |
+
# 🔥 Quantized Model: Mistral-Small-24B-Instruct-2501_GPTQ_G128_W4A16_MSE 🔥
|
| 20 |
|
| 21 |
This is a 4-bit quantized version of [mistralai/Mistral-Small-24B-Instruct-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501) model, quantized by [ConfidentialMind.com](https://www.confidentialmind.com) 🤖✨
|
| 22 |
It leverages the open-source GPTQModel quantization to achieve 4-bit precision with a group size of 128 resulting in a
|
|
|
|
| 26 |
Ran on a single NVIDIA A100 GPU with 80GB of VRAM.
|
| 27 |
|
| 28 |
*Note* `batch_size` is set quite high as the model is small, you may need to adjust this to your GPU VRAM.
|
| 29 |
+
*Note2* Due to the "packed" nature of mistral-small weights, MSE was used agressively along with a higher damping factor - this resulted in lesser loss and perplexity, however G32 is more advised
|
| 30 |
|
| 31 |
## Model Details
|
| 32 |
- **Original Model:** [mistralai/Mistral-Small-24B-Instruct-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501)
|