Release DeepSeek-V3-0324-IQ4_K_R4 and benchmarks

DeepSeek-V3-0324-IQ4_K_R4/DeepSeek-V3-0324-IQ4_K_R4-00001-of-00010.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eb8ef08b44a99223040cb02d2f89764eb03662669a65c690da670a3770521f57
+size 41169676352

DeepSeek-V3-0324-IQ4_K_R4/DeepSeek-V3-0324-IQ4_K_R4-00002-of-00010.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ddbd081cbdad380bb4548c81a2fc43a7f405d306f29678dfa1283b998c0ff3f
+size 42494252256

DeepSeek-V3-0324-IQ4_K_R4/DeepSeek-V3-0324-IQ4_K_R4-00003-of-00010.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:74789329b86ab85418f361e0e167c627ff94b0c12d27a1acd75823120c6b82e4
+size 42494252288

DeepSeek-V3-0324-IQ4_K_R4/DeepSeek-V3-0324-IQ4_K_R4-00004-of-00010.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7c2840f878709701a655caca5ee86952293cf00137677065582eed49595491a4
+size 42494252288

DeepSeek-V3-0324-IQ4_K_R4/DeepSeek-V3-0324-IQ4_K_R4-00005-of-00010.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a07cb7b0c4d8693fce701d08e9ec4cb2e693273279ba39fd17c3a1755439e81c
+size 42494252288

DeepSeek-V3-0324-IQ4_K_R4/DeepSeek-V3-0324-IQ4_K_R4-00006-of-00010.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:18483856dcc014e7aa32c55b641695ff05095822b86c05c87d901f9d1b3dfee2
+size 42494252288

DeepSeek-V3-0324-IQ4_K_R4/DeepSeek-V3-0324-IQ4_K_R4-00007-of-00010.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7c925b58c8394d1e965c930e2f6c415b0ea28cefb4bf6c383575f5e27d60c89a
+size 42494252288

DeepSeek-V3-0324-IQ4_K_R4/DeepSeek-V3-0324-IQ4_K_R4-00008-of-00010.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:520fecd53d32111018cd13c235d5731c737865497560726c4d253804476516ae
+size 42494252288

DeepSeek-V3-0324-IQ4_K_R4/DeepSeek-V3-0324-IQ4_K_R4-00009-of-00010.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8eba7dada84aad746661978ef4edcd6cf6b12d5a2cb27840d52d49dfeb89d882
+size 42494252288

DeepSeek-V3-0324-IQ4_K_R4/DeepSeek-V3-0324-IQ4_K_R4-00010-of-00010.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1af85ec57870ca34ea61152fd6ee2697bd8a3265006c8965ce80b12904ab1b46
+size 33542014112

README.md

@@ -4,15 +4,174 @@ pipeline_tag: text-generation
 base_model: deepseek-ai/DeepSeek-V3-0324
 license: mit
 base_model_relation: quantized
+tags:
+- mla
+- imatrix
+- deepseek_v3
+- conversational
 ---
 
-## `ik_llma.cpp` imatrix MLA Quantizations of DeepSeek-V3-0324 by deepseek-ai
+## `ik_llma.cpp` imatrix MLA Quantizations of DeepSeek-V3-0324
 
-This quant collection **REQUIRES** [ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp/) fork to support advanced non-linear SotA quants and Multi-Head Latent Attention (MLA). Do **not** download it and expect it to run on mainline vanilla llama.cpp, ollama, LM Studio, KoboldCpp, etc!
+This quant collection **REQUIRES** [ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp/) fork to support advanced non-linear SotA quants and Multi-Head Latent Attention (MLA). Do **not** download these big files and expect them to run on mainline vanilla llama.cpp, ollama, LM Studio, KoboldCpp, etc!
 
-These quants provide great perplexity for the size. MLA support allows 32k+ (or even 64k+) context length in under 24GB GPU VRAM for `R1` and `V3` while offloading MoE layers to RAM.
+These quants provide best in class perplexity for the given memory footprint. MLA support allows 32k+ context length in under 24GB GPU VRAM for `R1` and `V3` while offloading MoE layers to RAM.
 
-## imatrix
+Perfect for CPU+GPU systems with 24GB+ VRAM, and also CPU *only* rigs using dynamic quant repacking (for maximum memory throughput).
+
+You could try `ik_llama.cpp` quickly with your *existing* quants, as it computes MLA tensors and repacks quants on the fly at startup (if you have enough RAM+VRAM to fit entire model). Then come check out these fat quants here once you see the difference.
+
+## Big Thanks
+Shout out to Wendell and the **Level1Techs** crew, the community [Forums](https://forum.level1techs.com/t/deepseek-deep-dive-r1-at-home/225826), [YouTube Channel](https://www.youtube.com/@Level1Techs)!  **BIG thanks** for providing **BIG hardware** expertise and access to run these experiments and make these great quants available to the community!!!
+
+Also thanks to all the folks in the quanting and inferencing community here and on `r/LocalLLaMA` for tips and tricks helping each other run all the fun new models!
+
+Excited to share and learn together. Thanks!
+
+## Quant Collection
+So far these are my best recipes offering the lowest perplexity per GiB models suitable for a wide variety of CPU+GPU or CPU *only* rigs.
+
+#### `IQ4_K_R4` 4.936 BPW
+Special mix `IQ5_K_R4`/`IQ4_K_R4` routed experts with all other layers full `q8_0` for CPU+GPU offload or `--run-time-repack` for max speed CPU *only* rigs.
+Great for big 384+ GB RAM rig with 24GB+ GPU
+
+#### `IQ2_K_R4` 2.889 BPW
+Special mix `IQ3_K_R4`/`IQ2_K_R4` routed experts with all other layers full `q8_0` for CPU+GPU offload or `--run-time-repack` for max speed CPU *only* rigs.
+Great for CPU+GPU "troll rig" high end gamer systems e.g. 9950X 96 GB RAM + 3090TI 24 GB VRAM + Gen 5 NVMe SSD.
+
+#### Custom Mixes
+If you have multiple GPUs and more VRAM, you can make custom quants to optimize size and quants whatever hardware you have. If you have less VRAM, you could make a custom quant leaner in the non routed expert layers or get 64k+ context in 24GB VRAM. Also you can use the offline repack tool if you want to do CPU only with `mmap()` still enabled.
+
+## Quick Start
+#### `ik_llama.cpp` API server for GPU+CPU
+```bash
+# Fits 32k context in under 24GB VRAM
+# Optional `-ser 6,1` improves speed at minimal cost to quality
+CUDA_VISIBLE_DEVICES="0," \
+./build/bin/llama-server \
+    --model /mnt/raid/models/ubergarm/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324-IQ2_K_R4.gguf \
+    --alias ubergarm/DeepSeek-R1-V3-0324-IQ2_K_R4 \
+    --ctx-size 32768 \
+    -ctk q8_0 \
+    -mla 2 -fa \
+    -amb 512 \
+    -fmoe \
+    --temp 0.3 \
+    --min-p 0.05 \
+    --n-gpu-layers 63 \
+    --override-tensor exps=CPU \
+    --parallel 1 \
+    --threads 16 \
+    --host 127.0.0.1 \
+    --port 8080
+```
+
+#### `ik_llama.cpp` API server for CPU *only*
+```
+# The goal for now is as much RAM bandwidth in a single NUMA node e.g.
+# Use BIOS `NPS0` on AMD Epyc or single socket of Intel Xeon in BIOS `SNC=Disable`
+# Tune your `--threads` for token generation, and `--threads-batch` for prompt processing (prefill)
+# Note `--run-time-repack` will pre-allocate enough RAM for model weights instead of mmap()'ing off disk
+# Note there are options for both Explicit and Transparent Huge Pages with tuning discussions in [git repo](https://github.com/ikawrakow/ik_llama.cpp/pull/278#issuecomment-2746381515)
+numactl -N 0 -m 0 \
+./build/bin/llama-server \
+    --model /mnt/ai/models/ubergarm/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324-IQ4_K_R4.gguf \
+    --alias ubergarm/DeepSeek-V3-0324-IQ4_K_R4 \
+    --run-time-repack \
+    --ctx-size 65536 \
+    -ctk q8_0 \
+    -mla 3 -fa \
+    -amb 512 \
+    -fmoe \
+    --temp 0.3 \
+    --min-p 0.05 \
+    --parallel 1 \
+    --threads 88 \
+    --threads-batch 128 \
+    --numa numactl \
+    --host 127.0.0.1 \
+    --port 8080
+```
+
+## Quant Comparisons
+
+These are probably the **best quants available in this size class** for `V3-0324`!
+
+[!][Benchmarks showing these quants are smaller in size yet similar in performance to the `Q8_0`](benchmarks-01.png "Benchmarks showing these quants are smaller in size yet similar in performance to the `Q8_0`")
+
+ubergarm made no sacrifices for token embedding, attention, dense
+layers, or shared experts. This is possible because `ik_llama.cpp` MLA
+implementation saves so much GPU VRAM enabling 32k context in under 24GB
+VRAM. Also these quants use a new high quality imatrix including various
+coding samples and multiple written languages.  Routed expert layers
+make use of SotA CPU `IQx_K_R4` non-linear quants as well for likely
+best perplexity per GiB. 
+
+bartowski uses full token embedding quality but lower attention, dense
+layers, and shared expert quants. He does use a good quality imatrix with
+perplexity performance within the measurement error relative to this one.
+
+unsloth sacrifices token embedding with middle quality attention and
+dense layers, but no importance matrix.
+
+mradermacher modelcard side-bar is not showing so didn't yet fully
+compare exact recipe. Working with them to get info on their split GGUFs.
+
+#### Comparison Details
+
+<details>
+
+<summary>Detailed Comparison of ~Q2 Class Quants</summary>
+
+| | [ubergarm/DeepSeek-V3-0324-IQ2_K_R4](https://huggingface.co/ubergarm/DeepSeek-V3-0324-GGUF?show_file_info=DeepSeek-V3-0324-IQ2_K_R4%2FDeepSeek-V3-0324-IQ2_K_R4-00001-of-00005.gguf) | [bartowski/DeepSeek-V3-0324-Q2_K_L](https://huggingface.co/bartowski/deepseek-ai_DeepSeek-V3-0324-GGUF?show_file_info=deepseek-ai_DeepSeek-V3-0324-Q2_K_L%2Fdeepseek-ai_DeepSeek-V3-0324-Q2_K_L-00001-of-00007.gguf) | [unsloth/DeepSeek-V3-0324-UD-Q2_K_XL](https://huggingface.co/unsloth/DeepSeek-V3-0324-GGUF?show_file_info=UD-Q2_K_XL%2FDeepSeek-V3-0324-UD-Q2_K_XL-00001-of-00006.gguf) | [mradermacher/DeepSeek-V3-0324-i1-GGUF-Q2_K](https://huggingface.co/mradermacher/DeepSeek-V3-0324-i1-GGUF) |
+| --- | --- | --- | --- | --- |
+| **Overview**                       |            |        |        |        |
+| `split.tensors.count`              |  1147      |  1025  |  1025  |        |
+| `token_embd.weight`                | `Q8_0`     | `Q8_0` | `Q4_K` |        |
+| File Size (GiB)                    |   227      |   228  |   231  |        |
+| **Multi-Head Latent Attention**    |            |        |        |        |
+| `blk.*.attn_kv_b.weight`           | `Q8_0`     |   n/a  |   n/a  |   n/a  |
+| `blk.*.attn_k_b.weight`            | `Q8_0`     |   n/a  |   n/a  |   n/a  |
+| `blk.*.attn_v_b.weight`            | `Q8_0`     |   n/a  |   n/a  |   n/a  |
+| **Dense Layers**                   |            |        |        |        |
+| `blk.[0-2].attn_kv_a_mqa.weight`   | `Q8_0`     | `Q2_K` | `Q6_K` |        |
+| `blk.[0-2].attn_kv_a_norm.weight`  | `F32`      |  `F32` |  `F32` |        |
+| `blk.[0-2].attn_kv_b.weight`       | `Q8_0`     | `Q2_K` | `Q6_K` |        |
+| `blk.[0-2].attn_norm.weight`       | `F32`      |  `F32` |  `F32` |        |
+| `blk.[0-2].attn_q_a.weight`        | `Q8_0`     | `Q2_K` | `Q4_K` |        |
+| `blk.[0-2].attn_q_a_norm.weight`   | `F32`      |  `F32` |  `F32` |        |
+| `blk.[0-2].attn_q_b.weight`        | `Q8_0`     | `Q2_K` | `Q4_K` |        |
+| `blk.[0-2].ffn_down.weight`        | `Q8_0`     | `Q3_K` | `Q6_K` |        |
+| `blk.[0-2].ffn_gate.weight`        | `Q8_0`     | `Q2_K` | `Q4_K` |        |
+| `blk.[0-2].ffn_norm.weight`        | `F32`      |  `F32` |  `F32` |        |
+| `blk.[0-2].ffn_up.weight`          | `Q8_0`     | `Q2_K` | `Q4_K` |        |
+| `blk.[0-2].attn_output.weight`     | `Q8_0`     | `Q3_K` | `Q4_K` |        |
+| **Shared & Routed MoE Layers**     |            |        |        |        |
+| `blk.[3-60].attn_kv_a_mqa.weight`  | `Q8_0`     | `Q2_K` | `Q6_K` |        |
+| `blk.[3-60].attn_kv_a_norm.weight` | `F32`      | `F32`  | `F32`  |        |
+| `blk.[3-60].attn_kv_b.weight`      | `Q8_0`     | `Q2_K` | `Q6_K` |        |
+| `blk.[3-60].attn_norm.weight`      | `F32`      | `F32`  | `F32`  |        |
+| `blk.[3-60].attn_q_a.weight`       | `Q8_0`     | `Q2_K` | `Q4_K` |        |
+| `blk.[3-60].attn_q_a_norm.weight`  | `F32`      | `F32`  | `F32`  |        |
+| `blk.[3-60].attn_q_b.weight`       | `Q8_0`     | `Q2_K` | `Q4_K` |        |
+| `blk.[3-60].exp_probs_b.bias`      | `F32`      | `F32`  | `F32`  |        |
+| `blk.[3-60].ffn_down_exps.weight`  | `IQ3_K_R4` | `Q3_K` | `Q3_K` |        |
+| `blk.[3-60].ffn_down_shexp.weight` | `Q8_0`     | `Q3_K` | `Q6_K` |        |
+| `blk.[3-60].ffn_gate_exps.weight`  | `IQ2_K_R4` | `Q2_K` | `Q2_K` |        |
+| `blk.[3-60].ffn_gate_inp.weight`   | `F32`      | `F32`  | `F32`  |        |
+| `blk.[3-60].ffn_gate_shexp.weight` | `Q8_0`     | `Q2_K` | `Q4_K` |        |
+| `blk.[3-60].ffn_norm.weight`       | `F32`      | `F32`  | `F32`  |        |
+| `blk.[3-60].ffn_up_exps.weight`    | `IQ2_K_R4` | `Q2_K` | `Q2_K` |        |
+| `blk.[3-60].ffn_up_shexp.weight`   | `Q8_0`     | `Q2_K` | `Q4_K` |        |
+| `blk.[3-60].attn_output.weight`    | `Q8_0`     | `Q3_K` | `Q4_K` |        |
+| **Important Matrix & Perplexity**  |            |        |        |        |
+| `imatrix.dataset`                  | `calibration_data_v5_rc.txt`| `calibration_datav3.txt` | n/a | ? |
+| Final PPL (wiki.test.raw)          | 3.5614 +/- 0.02001  | ?      | ?  | ? |
+
+
+</details>
+
+#### imatrix
 
 <details>
 
@@ -59,33 +218,16 @@ Final estimate: PPL = 3.4755 +/- 0.03305
 
 </details>
 
-## Quant Collection
-
-#### `IQ2_K_R4` 
-Hybrid `IQ2_K_R4` non-linear quant for 32k context using `q8_0` MLA in for CPU+GPU offload with 96+GB RAM and 24+GB VRAM with minimal perplexity.
-
-<details>
-
-<summary>`IQ2_K_R4` Details Here</summary>
-
-```bash
-$ git branch
-* ik/make_qx_quants
-
-$ git rev-parse --short HEAD
-b9c25fe7
-```
-
----
-
-## Quantize Script
+#### Quant Cookers Secret Recipe
 
 ```bash
 #!/usr/bin/env bash
 
 custom="
-# Token embedding and output tensors (GPU)
+# Token embedding (GPU)
+# NOTE: cannot be a repacked type due to tensor size
 token_embd\.weight=q8_0
+# output tensors (GPU)
 output\.weight=q8_0
 output_norm\.weight=q8_0
 
@@ -93,6 +235,7 @@ output_norm\.weight=q8_0
 blk\.[0-2]\..*=q8_0
 
 # All attention, weights, and bias tensors for MoE layers (3-60) (GPU)
+# NOTE: attn_k_b.weight can't be k-, i-, or iqk-quant because its row size is 128
 blk\.[3-9]\.attn_.*=q8_0
 blk\.[1-5][0-9]\.attn_.*=q8_0
 blk\.60\.attn_.*=q8_0
@@ -114,7 +257,8 @@ blk\.[3-9]\.ffn_(gate|up)_shexp\.weight=q8_0
 blk\.[1-5][0-9]\.ffn_(gate|up)_shexp\.weight=q8_0
 blk\.60\.ffn_(gate|up)_shexp\.weight=q8_0
 
-# MoE Experts (3-60) (CPU)
+# Routed Experts (3-60) (CPU)
+# NOTE: Traditional wisdom suggests earlier layers use higher quants
 blk\.[3-9]\.ffn_down_exps\.weight=iq3_k_r4
 blk\.[1-5][0-9]\.ffn_down_exps\.weight=iq3_k_r4
 blk\.60\.ffn_down_exps\.weight=iq3_k_r4
@@ -140,9 +284,7 @@ custom=$(
     24
 ```
 
----
-
-## Perplexity
+#### Perplexity
 
 ```bash
 $ CUDA_VISIBLE_DEVICES="0," \
@@ -559,12 +701,10 @@ llama_print_timings:       total time = 2841519.57 ms / 287233 tokens
 Final estimate: PPL = 3.5614 +/- 0.02001
 ```
 
----
-
-## Split
+#### Split
 
 ```bash
-$ ./build/bin/llama-gguf-split 
+$ ./build/bin/llama-gguf-split \
     --dry-run \
     --split \
     --split-max-size 50G \
@@ -574,44 +714,6 @@ $ ./build/bin/llama-gguf-split
 
 </details>
 
-#### `TODO`
-
-- [ ] Upload good CPU *only* optimized inferencing quant
-
-## `ik_llama.cpp` API server
-
-```bash
-# I think temperature "1.0" on the API is 0.3 in llama.cpp ????
-# https://api-docs.deepseek.com/quick_start/parameter_settings
-# https://www.reddit.com/r/LocalLLaMA/comments/1jk0qjs/178bit_deepseekv30324_230gb_unsloth_dynamic_gguf/
-
-# Uses just under 24GB VRAM
-CUDA_VISIBLE_DEVICES="0," \
-./build/bin/llama-server \
-    --model /mnt/raid/models/ubergarm/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324-IQ2_K_R4.gguf \
-    --alias ubergarm/DeepSeek-R1-V3-0324-IQ2_K_R4 \
-    --ctx-size 32768 \
-    -ctk q8_0 \
-    -mla 2 -fa \
-    -amb 512 \
-    -fmoe \
-    --min-p 0.01 \
-    --temp 0.0 \
-    --n-gpu-layers 63 \
-    --override-tensor exps=CPU \
-    --parallel 1 \
-    --threads 16 \
-    --host 127.0.0.1 \
-    --port 8080
-```
-
-## Big Thanks
-Big thanks to all the folks in the quanting and inferencing community here and on `r/LocalLLaMA` for sharing tips and tricks to help each other access all the fun new models!
-
-Shout out to the **Level1Techs** crew, community [Forums](https://forum.level1techs.com/t/deepseek-deep-dive-r1-at-home/225826), [YouTube Channel](https://www.youtube.com/@Level1Techs), and for providing big hardware expertise and access to run these experiments!!!
-
-Finally, I'm still learning the ropes, so please be patient and we can learn together. Thanks!
-
 ## References
 * [ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp/)
 * [ik_llama.cpp Getting Started Guide](https://github.com/ikawrakow/ik_llama.cpp/discussions/258)