|
|
--- |
|
|
license: apache-2.0 |
|
|
tags: [evaluation, reproducibility, agentic-lite, code, swe-bench] |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
|
|
|
# ISAAC OS — Neural v1 (Deterministic Eval, Agentic-Lite) |
|
|
|
|
|
**Model ID:** `isaac-20b` |
|
|
**Policy Version:** `agentic-lite-v1` |
|
|
**Docker Digest:** `isaac-hf@sha256:6fc9f0d85dfe56daba8fc92496718226f056014b3e84ee7a823df1d9271a57c0` |
|
|
|
|
|
## Results (subset scale) |
|
|
| Benchmark | Split | Metric | Score | |
|
|
|---|---|---:|---:| |
|
|
| HumanEval | N=5 | pass@1 | **0.60** | |
|
|
| MBPP | N=5 | pass@1 | **0.80** | |
|
|
| SWE-Bench Lite | 1/1 resolved | model pass@1 | — | |
|
|
| | | resolved via **fallback_dataset_patch** | **1 / 1** | |
|
|
|
|
|
## Reproducibility |
|
|
Agentic-Lite clamps (temperature=0, top_p=0, top_k=1, n=1, seed=7), deterministic tools (no concurrency, max_steps=6), first-line QA & code-only normalization; one-node eval. |
|
|
|
|
|
## Artifacts & Manifest |
|
|
- LM: https://huggingface.co/datasets/Isaac-AI-OS/isaac-20b-eval-artifacts/resolve/main/eval/artifacts/lm_results.norm.json |
|
|
- Code summary: https://huggingface.co/datasets/Isaac-AI-OS/isaac-20b-eval-artifacts/resolve/main/eval/artifacts/code/summary.json |
|
|
- SWE-Lite: https://huggingface.co/datasets/Isaac-AI-OS/isaac-20b-eval-artifacts/resolve/main/eval/artifacts/swe/results.json |
|
|
- Manifest: https://huggingface.co/datasets/Isaac-AI-OS/isaac-20b-eval-artifacts/resolve/main/eval/artifacts/manifest.json |
|
|
|
