Accuracy Benchmarks¶

In srt-slurm, users can run different accuracy benchmarks by setting the benchmark section in the config yaml file. Supported benchmarks include mmlu, gpqa, longbenchv2, lm-eval, and AIME (via the script under configs/aime/).

Table of Contents¶

How Scoring Works
AIME
MMLU
GPQA
LongBench-V2
Configuration
Parameters
Available Categories
Example: Full Evaluation
Example: Quick Validation
Output
Important Notes
lm-eval (InferenceX)

Note: The context-length argument in the config yaml needs to be larger than the max_tokens argument of accuracy benchmark.

How Scoring Works¶

Accuracy benchmarks send a fixed dataset through the running OpenAI-compatible endpoint and compare each model response against the benchmark's expected answer. For AIME, NeMo Skills prompts the model to put the final answer in \boxed{...}, extracts that final boxed answer, and grades it with its math evaluator. There is no LLM judge in the default AIME path; the score is computed from exact/symbolic correctness.

When repeat is greater than 1, the benchmark runs multiple sampled generations per problem. NeMo Skills summarizes metrics across those generations, which is useful for comparing pass@1-style deterministic accuracy and sampled accuracy on the same serving setup.

AIME¶

AIME runs in the official NeMo Skills container (nvcr.io/nvidia/eval-factory/nemo-skills:26.03), side-by-side with the model server. There is no first-class type: aime runner — the eval logic lives in configs/aime/run.sh and recipes wire it up via type: custom.

Recipe shape¶

benchmark:
  type: custom
  container_image: nemo-skills    # alias defined in srtslurm.yaml `containers:`
                                  # or the full nvcr.io URI for Pyxis auto-pull
  env:
    OPENAI_API_KEY: "EMPTY"       # ns/litellm requires it set; value is unused
    HF_TOKEN: "${HF_TOKEN}"       # for gated HF datasets via ns prepare_data
    # Optional knob overrides — defaults match the upstream reasoning-eval reference:
    # MODEL: "dspro"           # must match served-model-name in the roles' args
    # DATASET: "aime25"        # aime24 | aime25 | aime26
    # REPEAT: "16"             # pass@k samples per problem
    # MAX_TOKENS: "400000"     # generous ceiling for reasoning traces
    # NUM_THREADS: "512"       # client-side concurrency
    # TEMPERATURE: "1.0"
    # TOP_P: "1.0"
    # SEED: "42"               # --starting_seed for reproducibility
  command: |
    bash /configs/aime/run.sh

Set the eval container alias in srtslurm.yaml:

containers:
  nemo-skills: "/shared/containers/nvidia+eval-factory+nemo-skills+26.03.sqsh"

Pre-cache the squashfs with: enroot import 'docker://nvcr.io#nvidia/eval-factory/nemo-skills:26.03'.

Reasoning-mode env vars (server side)¶

For reasoning-capable models (DeepSeek-V4-Pro thinking, GPT-OSS, etc.) — without these the model emits non-reasoning answers and AIME pass@k drops ~30 points below what the model can do.

roles:
  prefill:
    env:
      SGLANG_ENABLE_THINKING: "1"
      SGLANG_REASONING_EFFORT: "max"
  decode:
    env:
      SGLANG_ENABLE_THINKING: "1"
      SGLANG_REASONING_EFFORT: "max"

What the script does¶

ns prepare_data $DATASET — fetches the dataset into the NeMo Skills install.
ns eval ... against http://localhost:8000/v1 (the in-job dynamo frontend), pass@k=$REPEAT, with the upstream reasoning-eval reference's tuning defaults. NeMo Skills' default \boxed{} extractor scores the generations.

Outputs land at /logs/accuracy/<dataset>/eval-results/<dataset>/metrics.json with pass@1, pass@N, and majority@N.

Custom answer-extraction regex (not currently applied)¶

The SGLang team's reasoning-eval reference suggests broadening the answer extractor with these two ns eval overrides:

++eval_config.extract_from_boxed=False
++eval_config.extract_regex=(?:\boxed\{|\*\*Answer\*\*[^0-9\-]{0,30}|(?i:final answer)[^0-9\-]{0,30}|(?i:answer)\s*(?:is|=|:)[^0-9\-]{0,30})(-?\d+)

run.sh does not pass them. ns eval forwards Hydra ++overrides to parallel python -m nemo_skills.inference.generate subprocesses through nemo-run, which constructs the inner command line unquoted — bash strips backslashes from the regex before Python re.compile sees it, the regex becomes invalid, and every generate subprocess crashes on import. Verified on cluster runs that produced empty output dirs and a false "Benchmark completed successfully".

If you need a broader extractor for your model, post-process the cached output-rs<seed>.jsonl files with a Python script (raw-string regex, no shell layers).

MMLU¶

For MMLU dataset, the benchmark section in yaml file can be modified in the following way:

benchmark:
  type: "mmlu"
  num_examples: 200 # Number of examples to run
  max_tokens: 2048 # Max number of output tokens
  repeat: 8 # Number of repetition
  num_threads: 512 # Number of parallel threads for running benchmark

Then launch the script as usual:

srtctl apply -f config.yaml

After finishing benchmarking, the benchmark.out will contain the results of accuracy:

====================
Repeat: 8, mean: 0.812
Scores: ['0.790', '0.820', '0.800', '0.820', '0.820', '0.790', '0.820', '0.840']
====================
Writing report to /tmp/mmlu_deepseek-ai_DeepSeek-R1.html
{'other': np.float64(0.9), 'other:std': np.float64(0.30000000000000004), 'score:std': np.float64(0.36660605559646725), 'stem': np.float64(0.8095238095238095), 'stem:std': np.float64(0.392676726249301), 'humanities': np.float64(0.7428571428571429), 'humanities:std': np.float64(0.4370588154508102), 'social_sciences': np.float64(0.9583333333333334), 'social_sciences:std': np.float64(0.19982631347136331), 'score': np.float64(0.84)}
Writing results to /tmp/mmlu_deepseek-ai_DeepSeek-R1.json
Total latency: 465.618 s
Score: 0.840
Results saved to: /logs/accuracy/mmlu_deepseek-ai_DeepSeek-R1.json
MMLU evaluation complete

GPQA¶

For GPQA dataset, the benchmark section in yaml file can be modified in the following way:

benchmark:
  type: "gpqa"
  num_examples: 198 # Number of examples to run
  max_tokens: 65536 # We need a larger output token number for GPQA
  repeat: 8 # Number of repetition
  num_threads: 128 # Number of parallel threads for running benchmark

The context-length argument here should be set to a value larger than max_tokens.

LongBench-V2¶

LongBench-V2 is a long-context evaluation benchmark that tests model performance on extended context tasks. It's particularly useful for validating models with large context windows (128K+ tokens).

Configuration¶

benchmark:
  type: "longbenchv2"
  max_context_length: 128000  # Maximum context length (default: 128000)
  num_threads: 16             # Concurrent evaluation threads (default: 16)
  max_tokens: 16384           # Maximum output tokens (default: 16384)
  num_examples: 100           # Number of examples to run (default: all)
  categories:                 # Task categories to evaluate (default: all)
    - "single_doc_qa"
    - "multi_doc_qa"
    - "summarization"
    - "few_shot_learning"
    - "code_completion"
    - "synthetic"

Parameters¶

Parameter	Type	Default	Description
`max_context_length`	int	128000	Maximum context length for evaluation. Should not exceed model's trained context window.
`num_threads`	int	16	Number of concurrent threads for parallel evaluation. Increase for faster throughput on high-capacity endpoints.
`max_tokens`	int	16384	Maximum tokens for model output. Must be less than `context-length` in the roles' `args`.
`num_examples`	int	all	Limit the number of examples to evaluate. Useful for quick validation runs.
`categories`	list	all	Specific task categories to run. Omit to run all categories.

Available Categories¶

LongBench-V2 includes the following task categories:

single_doc_qa: Single document question answering
multi_doc_qa: Multi-document question answering
summarization: Long document summarization
few_shot_learning: Few-shot learning with long context
code_completion: Long-context code completion
synthetic: Synthetic long-context tasks (needle-in-haystack, etc.)

Example: Full Evaluation¶

Run complete LongBench-V2 evaluation with all categories:

schema: 2
name: "longbench-v2-eval"

model:
  path: "deepseek-r1"
  container: "latest"
  precision: "fp8"

resources:
  gpu_type: "gb200"
  gpus_per_node: 4

engine: sglang
roles:
  prefill:
    nodes: 2
    workers: 2
    args:
      context-length: 131072  # Must exceed max_tokens
      tensor-parallel-size: 4
  decode:
    nodes: 4
    workers: 2
    args:
      context-length: 131072
      tensor-parallel-size: 8

benchmark:
  type: "longbenchv2"
  max_context_length: 128000
  max_tokens: 16384
  num_threads: 32

Example: Quick Validation¶

Run a quick subset for validation:

benchmark:
  type: "longbenchv2"
  num_examples: 50           # Limit to 50 examples
  num_threads: 8
  categories:
    - "single_doc_qa"        # Only run single-doc QA

Output¶

After completion, results are saved to the logs directory:

/logs/accuracy/longbenchv2_<model_name>.json

The output includes per-category scores and aggregate metrics:

{
  "model": "deepseek-ai/DeepSeek-R1",
  "scores": {
    "single_doc_qa": 0.82,
    "multi_doc_qa": 0.78,
    "summarization": 0.85,
    "few_shot_learning": 0.76,
    "code_completion": 0.81,
    "synthetic": 0.92
  },
  "overall_score": 0.82,
  "total_examples": 500,
  "total_latency_s": 1842.5
}

Important Notes¶

Context Length: Ensure context-length in your roles' args exceeds max_tokens for the benchmark
Memory: Long-context evaluation requires significant GPU memory. Use appropriate mem-fraction-static settings
Throughput: Increase num_threads for faster evaluation, but monitor for OOM errors
Categories: Running specific categories is useful for targeted validation (e.g., just testing summarization capabilities)

lm-eval (InferenceX)¶

The lm-eval benchmark runner integrates EleutherAI/lm-evaluation-harness via InferenceX's benchmark_lib.sh. Unlike the built-in benchmarks above, this runner sources evaluation logic from an external InferenceX workspace mounted at /infmax-workspace.

This is used by InferenceX CI to run evals such as GSM8K and GPQA against NVIDIA multi-node disaggregated deployments on GB200, GB300, B200, B300, H100, and H200. AMD MI355X multi-node evals are handled by InferenceX's upstreamed AMD Slurm path, not by this srt-slurm runner.

In InferenceX CI, recipes normally keep their throughput benchmark configuration. do_sweep.py invokes the registered lm-eval runner as a post-step when RUN_EVAL=true, or as the only benchmark-like step when EVAL_ONLY=true. There is no separate infmax-eval benchmark type.

How it works¶

RuntimeContext mounts the host path from INFMAX_WORKSPACE at /infmax-workspace inside the Slurm container.
do_sweep.py starts infrastructure, workers, and the frontend for the normal recipe topology.
For EVAL_ONLY=true, do_sweep.py skips the throughput benchmark stage and runs _run_post_eval() directly after frontend startup.
_run_post_eval() waits for the OpenAI-compatible endpoint on port 8000 and, in eval-only mode, performs the full wait_for_model() health check for the configured prefill/decode or aggregated topology.
_run_post_eval() launches the registered lm-eval runner on the head node and passes through InferenceX metadata such as framework, precision, sequence length, prefill/decode topology, and eval concurrency.
The runner script (benchmarks/scripts/lm-eval/bench.sh) uses MODEL_NAME from do_sweep.py, or auto-discovers the served model from /v1/models as a fallback.
The runner sources /infmax-workspace/benchmarks/benchmark_lib.sh, runs run_eval --framework lm-eval, and calls append_lm_eval_summary.
Eval artifacts are copied to /logs/eval_results/ for InferenceX launcher-side artifact pickup.

EVAL_ONLY mode¶

srt-slurm supports an EVAL_ONLY mode for CI jobs that should only validate accuracy. This is controlled by environment variables from the InferenceX workflow:

Env var	Description
`EVAL_ONLY`	Set to `true` to skip the throughput benchmark stage and run eval only
`RUN_EVAL`	Set to `true` to run eval after the throughput benchmark completes
`EVAL_CONC`	Concurrent requests for lm-eval, normally set by InferenceX from the generated `eval-conc` value
`INFMAX_WORKSPACE`	Host path to the InferenceX checkout that should be mounted at `/infmax-workspace`
`MODEL_NAME`	Served model alias for OpenAI-compatible requests; set by `do_sweep.py` from `config.served_model_name`

When EVAL_ONLY=true: - Stage 4 skips the throughput benchmark entirely. No throughput result JSON is expected from srt-slurm. - The eval path uses the full wait_for_model() health check before starting lm-eval. - _run_post_eval() launches the lm-eval runner and returns its exit code. - Eval failure is fatal because eval is the only purpose of the job.

When RUN_EVAL=true (without EVAL_ONLY): - Throughput benchmark runs normally - After benchmark completes successfully, eval runs as a post-step - Eval failure is non-fatal; the benchmark job still succeeds if throughput passed

Environment variables¶

The following env vars are passed through to the lm-eval runner container:

Env var	Purpose
`RUN_EVAL`, `EVAL_ONLY`, `IS_MULTINODE`	Control whether eval runs and how InferenceX classifies the artifact
`FRAMEWORK`, `PRECISION`, `MODEL_PREFIX`, `RUNNER_TYPE`, `SPEC_DECODING`	Benchmark identity metadata for `meta_env.json`
`ISL`, `OSL`, `RESULT_FILENAME`	Sequence length and result-file metadata
`MODEL`, `MODEL_PATH`, `MODEL_NAME`	Model metadata and the served model alias used for requests
`MAX_MODEL_LEN`, `EVAL_MAX_MODEL_LEN`	Context-length metadata used by InferenceX eval helpers when available
`PREFILL_TP`, `PREFILL_EP`, `PREFILL_NUM_WORKERS`, `PREFILL_DP_ATTN`	Prefill-side topology metadata
`DECODE_TP`, `DECODE_EP`, `DECODE_NUM_WORKERS`, `DECODE_DP_ATTN`	Decode-side topology metadata
`EVAL_CONC`, `EVAL_CONCURRENT_REQUESTS`	Eval concurrency controls

The runner maps srt-slurm's PREFILL_DP_ATTN and DECODE_DP_ATTN names to InferenceX's PREFILL_DP_ATTENTION and DECODE_DP_ATTENTION names before calling append_lm_eval_summary. This is required for multi-node summary tables to preserve prefill/decode DPA state.

Concurrency¶

Eval concurrency is ultimately read by InferenceX's benchmark_lib.sh from EVAL_CONCURRENT_REQUESTS. The runner script sets that value from EVAL_CONC when present, preserves an existing EVAL_CONCURRENT_REQUESTS otherwise, and falls back to 256 only if neither variable is set:

export EVAL_CONCURRENT_REQUESTS="${EVAL_CONC:-${EVAL_CONCURRENT_REQUESTS:-256}}"

The InferenceX workflow sets EVAL_CONC from the generated eval-conc value. For multi-node configs, InferenceX selects the 8k1k entry with the highest max eligible concurrency for each (model, runner, framework, precision, spec-decoding, prefill-dp-attn, decode-dp-attn) group, then sets eval-conc to the upper median of that config's eligible concurrency list. If EVAL_CONC is not set in the environment, do_sweep.py falls back to the max of the recipe benchmark concurrency list.

Output¶

Eval artifacts are written to /logs/eval_results/ inside the container: - meta_env.json - metadata used by InferenceX aggregation and summary tables - results*.json - lm-eval scores per task - sample*.jsonl - per-sample outputs

These are collected by the InferenceX NVIDIA launch scripts and uploaded as workflow artifacts. In eval-only mode the InferenceX workflow expects eval artifacts, not throughput benchmark artifacts.

Intricacies¶

Eval floor of 16
There is 1 sweep config of conc: [1], which causes evals to take >4hrs to complete.