Evaluation is All You Need: Strategic Overclaiming of LLM Reasoning Capabilities Through Evaluation Design
lincharliesun
Abstract
Empirical assessments reveal significant fluctuations in benchmark evaluation results of Deepseek-R1-Distill models, questioning the reliability of claimed performance improvements and advocating for a more rigorous evaluation paradigm.
Reasoning models represented by the Deepseek-R1-Distill series have been widely adopted by the open-source community due to their strong performance in mathematics, science, programming, and other domains. However, our study reveals that their benchmark evaluation results are subject to significant fluctuations caused by various factors. Subtle differences in evaluation conditions can lead to substantial variations in results. Similar phenomena are observed in other open-source inference models fine-tuned based on the Deepseek-R1-Distill series, as well as in the QwQ-32B model, making their claimed performance improvements difficult to reproduce reliably. Therefore, we advocate for the establishment of a more rigorous paradigm for model performance evaluation and present our empirical assessments of the Deepseek-R1-Distill series models.
Community
Reasoning models represented by the Deepseek-R1-Distill series have been widely adopted by the open-source community due to their strong performance in mathematics, science, programming, and other domains. The study reveals that their benchmark evaluation results are subject to significant fluctuations caused by various factors. Subtle differences in evaluation conditions can lead to substantial variations in results. Similar phenomena are observed in other open-source inference models fine-tuned based on the Deepseek-R1-Distill series, as well as in the QwQ-32B model, making their claimed performance improvements difficult to reproduce reliably.
What are the main takeaways?
This study reveals how minor variations in evaluation design —such as dataset versions, seed initialization, instruction placement, option ordering, and Tensor Parallelism—can cause significant fluctuations in benchmark scores for reasoning-focused LLMs like the Deepseek-R1-Distill series.
Key Findings:
- Evaluation Conditions Matter : Small changes can lead to performance swings of several percentage points, making model comparisons unreliable.
- Seed Initialization & N-Sampling : Results are highly sensitive to random seeds. Using more inference samples (larger N) improves stability.
- Dataset Versions : Differences in image handling or formatting (e.g., AIME datasets) affect results by up to 3.9 percentage points .
- Option & Answer Bias : In MCQs like GPQA Diamond, answer position and option order can reduce accuracy by over 5 percentage points .
- Instruction Placement : Minor impact but still affects reproducibility depending on training alignment.
Issues Identified:
Many open-source models overstate performance gains due to favorable evaluation setups , not actual improvements.
Lack of transparency and standardization leads to unreproducible results .
Recommendations:
- Use fixed seeds , clear documentation , and confidence intervals .
- Report average performance with statistical stability , not just peak scores.
- Adopt standardized evaluation frameworks and disclose all settings.
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper