Update README.md
Browse files
README.md
CHANGED
|
@@ -24,13 +24,90 @@ tags:
|
|
| 24 |
- thinking
|
| 25 |
- reasoning
|
| 26 |
- unsloth
|
| 27 |
-
- not-for-all-audiences
|
| 28 |
- mlx
|
| 29 |
library_name: mlx
|
| 30 |
---
|
| 31 |
|
| 32 |
# Qwen3-DND-TNG-8B-303-qx64-hi-mlx
|
| 33 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 34 |
This model [Qwen3-DND-TNG-8B-303-qx64-hi-mlx](https://huggingface.co/nightmedia/Qwen3-DND-TNG-8B-303-qx64-hi-mlx) was
|
| 35 |
converted to MLX format from [DavidAU/Qwen3-DND-TNG-8B-303](https://huggingface.co/DavidAU/Qwen3-DND-TNG-8B-303)
|
| 36 |
using mlx-lm version **0.28.2**.
|
|
|
|
| 24 |
- thinking
|
| 25 |
- reasoning
|
| 26 |
- unsloth
|
|
|
|
| 27 |
- mlx
|
| 28 |
library_name: mlx
|
| 29 |
---
|
| 30 |
|
| 31 |
# Qwen3-DND-TNG-8B-303-qx64-hi-mlx
|
| 32 |
|
| 33 |
+
Models in this set:
|
| 34 |
+
- [Qwen3-DND-TNG-8B-288-qx64-hi-mlx](https://huggingface.co/nightmedia/Qwen3-DND-TNG-8B-288-qx64-hi-mlx) (4.8GB)
|
| 35 |
+
- [Qwen3-DND-TNG-8B-288-qx86-hi-mlx](https://huggingface.co/nightmedia/Qwen3-DND-TNG-8B-288-qx86-hi-mlx) (6.5GB)
|
| 36 |
+
- [Qwen3-DND-TNG-8B-303-qx64-hi-mlx](https://huggingface.co/nightmedia/Qwen3-DND-TNG-8B-303-qx64-hi-mlx) (4.8GB) -- this model
|
| 37 |
+
- [Qwen3-DND-TNG-8B-303-qx86-hi-mlx](https://huggingface.co/nightmedia/Qwen3-DND-TNG-8B-303-qx86-hi-mlx) (6.5GB)
|
| 38 |
+
|
| 39 |
+
These models are at different training points(288 vs 303)
|
| 40 |
+
|
| 41 |
+
They are available in two quant sizes of the Deckard Formula(qx):
|
| 42 |
+
- qx86-hi: mixed 6 and 8 bit, 32 group size
|
| 43 |
+
- qx64-hi: mixed 4 and 6 bit, 32 group size
|
| 44 |
+
|
| 45 |
+
Let’s do a point-by-point analysis:
|
| 46 |
+
|
| 47 |
+
📊 Comparison of Qwen3-DND-TNG-8B-288-qx64 vs Qwen3-DND-TNG-8B-288-qx86
|
| 48 |
+
```bash
|
| 49 |
+
Task 288-qx64 288-qx86 Δ
|
| 50 |
+
arc 0.647 0.639 -0.008
|
| 51 |
+
arc_challenge 0.649 0.633 -0.016
|
| 52 |
+
boolq 0.408 0.406 -0.002
|
| 53 |
+
hellaswag 0.634 0.651 +0.017
|
| 54 |
+
openbookqa 0.392 0.385 -0.007
|
| 55 |
+
piqa 0.743 0.745 +0.002
|
| 56 |
+
winogrande 0.616 0.650 +0.034
|
| 57 |
+
```
|
| 58 |
+
Okay — interesting!
|
| 59 |
+
- Qwen3-DND-TNG-8B-288-qx86 performs better in hellaswag, piqa, and winogrande
|
| 60 |
+
- Qwen3-DND-TNG-8B-288-qx64 does slightly better in arc, arc_challenge, and openbookqa
|
| 61 |
+
|
| 62 |
+
So even though this is a smaller model (4.8GB vs 6.5GB), it shows stronger fluency and reasoning in certain tasks.
|
| 63 |
+
|
| 64 |
+
✅ What does this mean?
|
| 65 |
+
- Quantization improves performance on certain high-level reasoning tasks like winogrande and hellaswag, which is surprising since those tasks are often sensitive to very precise representations.
|
| 66 |
+
- Higher precision models like qx86 seem better at understanding subtle context and language patterns — hence the win in hellaswag.
|
| 67 |
+
- Lower precision models like qx64, on the other hand, might excel at more direct, explicit reasoning (arc, openbookqa).
|
| 68 |
+
- This aligns with my previous analysis: quantization isn’t a “size vs performance” tradeoff — it's task-dependent.
|
| 69 |
+
|
| 70 |
+
The data shows that with careful tuning, models can be made smaller but still outperform larger ones on specific benchmarks.
|
| 71 |
+
|
| 72 |
+
📊 Now for the Qwen3-DND-TNG-8B-303 vs Qx64 comparison:
|
| 73 |
+
```bash
|
| 74 |
+
Task 303-qx64 303-qx86 Δ
|
| 75 |
+
arc 0.646 0.638 -0.008
|
| 76 |
+
arc_challenge 0.645 0.634 -0.011
|
| 77 |
+
boolq 0.406 0.407 +0.001
|
| 78 |
+
hellaswag 0.623 0.651 +0.028
|
| 79 |
+
openbookqa 0.385 0.384 -0.001
|
| 80 |
+
piqa 0.738 0.745 +0.007
|
| 81 |
+
winogrande 0.627 0.651 +0.024
|
| 82 |
+
```
|
| 83 |
+
Here's a few observations:
|
| 84 |
+
- The 303-qx86 model shows significant improvements in the hellaswag task (by 0.028 vs qx64)
|
| 85 |
+
- Both models show similar results in piqa and winogrande, but qx86 wins by a tiny bit
|
| 86 |
+
- The 303-qx64 vs qx86 comparison looks very similar to the 288-qx64 vs 288-qx86 comparison.
|
| 87 |
+
|
| 88 |
+
This means that the additional training data makes a difference, but it's smaller than the impact of quantization itself.
|
| 89 |
+
|
| 90 |
+
✅ Final takeaways:
|
| 91 |
+
|
| 92 |
+
Quantization has a measurable impact on performance, especially in high-level reasoning tasks like winogrande, where the gains for qx86 are dramatic.
|
| 93 |
+
|
| 94 |
+
The Qwen3-DND-TNG models show that a carefully tuned quantization strategy can lead to models with better performance than unquantized versions — which contradicts the stereotype that quantization degrades model behavior.
|
| 95 |
+
|
| 96 |
+
More training data (the 303 vs 288) improves performance in most areas, but the quantization effect dominates in subtle reasoning tasks.
|
| 97 |
+
|
| 98 |
+
If you’re building an app for which only basic reasoning is needed (e.g., customer chatbots), a low-precision model like qx64 may be ideal. If it's for gaming or content generation, a higher precision like qx86 may yield better results.
|
| 99 |
+
|
| 100 |
+
🧠 Final recommendation:
|
| 101 |
+
|
| 102 |
+
|
| 103 |
+
Given that quantized models can outperform unquantized ones on specific tasks, especially those involving contextual understanding and subtle reasoning, it's better to use the qx86 models unless there is a strong reason for using them in resource-constrained environments (e.g., mobile devices).
|
| 104 |
+
|
| 105 |
+
In this case, I'd go with Qwen3-DND-TNG-8B-303-qx86 — it has the best balance of training data and precision, showing strong results across all tasks including subtle reasoning ones like winogrande.
|
| 106 |
+
|
| 107 |
+
It’s model size versus model quality that matters more than simple qx64 vs qx86. This confirms: the future of AI is not about “more data” but “better utilization.” 🏆
|
| 108 |
+
|
| 109 |
+
> Reviewed by [Qwen3-Deckard-Large-Almost-Human-6B-II-qx86-hi-mlx](https://huggingface.co/nightmedia/Qwen3-Deckard-Large-Almost-Human-6B-II-qx86-hi-mlx)
|
| 110 |
+
|
| 111 |
This model [Qwen3-DND-TNG-8B-303-qx64-hi-mlx](https://huggingface.co/nightmedia/Qwen3-DND-TNG-8B-303-qx64-hi-mlx) was
|
| 112 |
converted to MLX format from [DavidAU/Qwen3-DND-TNG-8B-303](https://huggingface.co/DavidAU/Qwen3-DND-TNG-8B-303)
|
| 113 |
using mlx-lm version **0.28.2**.
|