# OpenSeek-Small-v1-Baseline Model Documentation

## Overview
We sampled 100 billion tokens from the CCI4.0 dataset and trained a 1.4B-parameter MoE model with 0.4B active parameters. This model, along with the dataset, is open-sourced as a baseline for future experiments in areas such as dataset construction, algorithmic strategies, and parallel training frameworks. The model arch is same as OpenSeek-Small-v1 model.

## Training Data
The ratio for each domain is as follows:

| Name                                      | Ratio   |
|-------------------------------------------|---------|
| Nemotron-CC-high-actual-actual-high       | 1.1068  |
| Nemotron-CC-high-actual-actual-low        | 0.3577  |
| Nemotron-CC-high-actual-actual-mid        | 0.7775  |
| Nemotron-CC-high-synthetic-distill-high   | 0.2859  |
| Nemotron-CC-high-synthetic-distill-low    | 0.1672  |
| Nemotron-CC-high-synthetic-distill-mid    | 0.2339  |
| Nemotron-CC-high-synthetic-diverse_qa_pairs-high | 0.5397  |
| Nemotron-CC-high-synthetic-diverse_qa_pairs-low  | 0.4064  |
| Nemotron-CC-high-synthetic-diverse_qa_pairs-mid  | 0.5005  |
| Nemotron-CC-high-synthetic-extract_knowledge-high | 0.4616  |
| Nemotron-CC-high-synthetic-extract_knowledge-low  | 0.0670  |
| Nemotron-CC-high-synthetic-extract_knowledge-mid  | 0.3429  |
| Nemotron-CC-high-synthetic-knowledge_list-high | 0.2610  |
| Nemotron-CC-high-synthetic-knowledge_list-low  | 0.1824  |
| Nemotron-CC-high-synthetic-knowledge_list-mid  | 0.2313  |
| Nemotron-CC-high-synthetic-wrap_medium-high | 0.8237  |
| Nemotron-CC-high-synthetic-wrap_medium-low  | 0.2866  |
| Nemotron-CC-high-synthetic-wrap_medium-mid  | 0.6670  |
| Nemotron-CC-low-synthetic-wrap_medium-high | 0.4657  |
| Nemotron-CC-low-synthetic-wrap_medium-low  | 0.2005  |
| Nemotron-CC-low-synthetic-wrap_medium-mid  | 0.4317  |
| Nemotron-CC-medium-actual-actual-high      | 1.1397  |
| Nemotron-CC-medium-actual-actual-low       | 0.6782  |
| Nemotron-CC-medium-actual-actual-mid       | 0.9175  |
| arxiv                                    | 0.6414  |
| books                                    | 0.4696  |
| code-high                                | 1.0102  |
| code-low                                 | 1.1403  |
| code-mid                                 | 0.9674  |
| cot_synthesis2_CC-high                   | 0.3755  |
| cot_synthesis2_CC-low                    | 0.0499  |
| cot_synthesis2_CC-mid                    | 1.8299  |
| cot_synthesis2_OpenSource-high           | 0.2573  |
| cot_synthesis2_OpenSource-low            | 0.1638  |
| cot_synthesis2_OpenSource-mid            | 0.3251  |
| cot_synthesis2_arxiv-high                | 6.0237  |
| cot_synthesis2_arxiv-low                 | 8.9063  |
| cot_synthesis2_arxiv-mid                 | 10.1376 |
| cot_synthesis2_code-high                 | 0.4598  |
| cot_synthesis2_code-low                  | 0.6857  |
| cot_synthesis2_code-mid                  | 0.8990  |
| cot_synthesis2_math-high                 | 1.3135  |
| cot_synthesis2_math-low                  | 1.6530  |
| cot_synthesis2_math-mid                  | 0.3536  |
| cot_synthesis2_wiki-high                 | 0.6314  |
| cot_synthesis2_wiki-low                  | 0.5978  |
| cot_synthesis2_wiki-mid                  | 0.7909  |
| cot_synthesis_CC-high                    | 0.2225  |
| cot_synthesis_CC-low                     | 0.1797  |
| cot_synthesis_CC-mid                     | 0.2042  |
| cot_synthesis_OpenSource-high            | 0.4081  |
| cot_synthesis_OpenSource-low             | 0.1659  |
| cot_synthesis_OpenSource-mid             | 1.2828  |
| cot_synthesis_arxiv-high                 | 5.68    |
| cot_synthesis_arxiv-low                  | 7.4907  |
| cot_synthesis_arxiv-mid                  | 8.9359  |
| cot_synthesis_code-high                  | 0.7663  |
| cot_synthesis_code-low                   | 0.4052  |
| cot_synthesis_code-mid                   | 0.1916  |
| cot_synthesis_math-high                  | 0.5074  |
| cot_synthesis_math-low                   | 0.6437  |
| cot_synthesis_math-mid                   | 0.6406  |
| cot_synthesis_wiki-high                  | 0.4000  |
| cot_synthesis_wiki-low                   | 0.3564  |
| cot_synthesis_wiki-mid                   | 0.5768  |
| math-high                                | 1.8165  |
| math-low                                 | 1.6940  |
| math-mid                                 | 1.6311  |
| pes2o                                    | 6.1982  |
| pes2o-full-train                         | 1.4257  |
| pes2o-full-val                           | 0.0143  |
| stack                                    | 0.4229  |
| wiki                                     | 0.4202  |
| zh_cc-high-loss0                         | 1.8171  |
| zh_cc-high-loss1                         | 0.9776  |
| zh_cc-high-loss2                         | 0.3725  |
| zh_cc-medium-loss0                       | 0.9492  |
| zh_cc-medium-loss1                       | 0.9236  |
| zh_cc-medium-loss2                       | 1.0643  |

## Wandb
Our training curves have been recorded in Weights & Biases [wandb](https://wandb.ai/openseek-team/OpenSeek-Small-v1-Baseline).

## Evalation
We used the LightEval library for model evaluation, following the same setup as in FineWeb and CCI3-HQ. 
All evaluations were conducted in a zero-shot setting. To directly compare the performance across different datasets, we use Average, which refers to the overall average score across all Chinese and English benchmarks. 
| Metrics           | Score   |
|-------------------|---------|
| HellaSwag         | 42.09   |
| ARC (Average)     | 40.11   |
| PIQA              | 67.14   |
| MMLU (cloze)      | 31.29   |
| CommonsenseQA     | 28.17   |
| TriviaQA          | 6.51    |
| Winograde         | 51.38   |
| OpenBookQA        | 33.00   |
| GSM8K (5-shot)    | 6.67    |
| SIQA              | 41.86   |
| CEval             | 30.19   |
| CMMLU             | 30.25   |
| **Average-English** | **34.82** |
| **Average-Chinese** | **30.22** |
| **Overall Average** | **32.52** |

## Usage Instructions
```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("BAAI/OpenSeek-Small-v1-Baseline")
tokenizer = AutoTokenizer.from_pretrained("BAAI/OpenSeek-Small-v1-Baseline")

inputs = tokenizer("The future of AI is", return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
print(tokenizer.decode(outputs[0]))
```