Performance Benchmarks#
Platforms: NVIDIA Jetson AGX Thor Developer Kit, Jetson AGX Orin 64GB, Jetson Orin NX 16GB, Jetson Orin Nano 8GB, and NVIDIA DGX Spark (GB10). Older release sections may include a narrower platform set.
Definitions#
Term |
Description |
|---|---|
Prefill time |
Average wall-clock time (ms) to process the input prompt |
Prefill throughput |
Prompt tokens processed per second during prefill (tok/s) |
Generation throughput |
Tokens generated per second during decoding (tok/s) |
Batch size |
Number of concurrent sequences (BS=1 = single-user latency, BS=8 = multi-user throughput) |
Acceptance rate |
Average tokens accepted per speculative decoding verify step (higher is better) |
Speedup |
Speculative decoding generation throughput / vanilla generation throughput (same model, precision, batch size) |
ViT time |
Total visual encoder processing time per inference run (ms) |
ViT throughput |
Image tokens processed per second by the visual encoder (tok/s) |
GPU memory |
Peak GPU memory usage during inference (MB) |
MTP |
Multi-token prediction speculative decoding |
DFlash |
z-lab paired-draft speculative decoding with a dedicated external draft checkpoint |
JetSpec |
JetSpec paired-draft speculative decoding with causal proposal attention and branching tree verification |
Precision Key#
Precision |
Description |
Platform Requirement |
|---|---|---|
FP16 |
Half-precision float |
All platforms |
FP8 |
8-bit float |
SM89+ (Ada Lovelace and newer) |
INT4 AWQ |
4-bit integer (AWQ quantization) |
All platforms |
INT4 GPTQ |
4-bit integer (GPTQ quantization) |
All platforms |
NVFP4 |
NVIDIA 4-bit float |
SM100+ (Blackwell and newer) |
Reproducing Benchmark Runs#
Use the public TensorRT Edge-LLM user guides for model export, engine build, and basic inference:
Quick Start Guide for LLM and VLM export, component engine builds,
llm_inference, andllm_bench.Speculative Decoding for EAGLE3, MTP, DFlash, and JetSpec export/build layouts.
Input Format Guide for the Edge-LLM JSON request format and chat-template behavior.
This section records the benchmark-specific dataset choices, build-time limits,
runtime flags, and llm_bench shapes used to reproduce the tables.
1. Prepare Datasets#
Generate datasets with the scripts under examples/accuracy/scripts. MTBench
and MMLU are text-only LLM datasets. COCO and MMMU are VLM datasets.
python3 -m pip install -r examples/accuracy/requirements-datasets.txt
DATASET_DIR=/path/to/datasets
mkdir -p "${DATASET_DIR}"
# LLM generation benchmark dataset.
python3 examples/accuracy/scripts/prepare_dataset.py \
--dataset MTBench \
--output_dir "${DATASET_DIR}/mtbench" \
--batch_size 1
# LLM multiple-choice accuracy-side dataset.
python3 examples/accuracy/scripts/prepare_dataset.py \
--dataset MMLU \
--output_dir "${DATASET_DIR}/mmlu" \
--batch_size 1
# VLM generation benchmark dataset.
python3 examples/accuracy/scripts/prepare_dataset.py \
--dataset COCO \
--output_dir "${DATASET_DIR}/coco" \
--batch_size 1
# VLM multiple-choice accuracy-side dataset.
python3 examples/accuracy/scripts/prepare_dataset.py \
--dataset MMMU \
--output_dir "${DATASET_DIR}/mmmu" \
--batch_size 1
2. Export and Build Specs#
Follow the linked export/build docs for the selected model family, precision, and decoding mode. Use these benchmark-specific build parameters:
Engine |
|
|---|---|
Vanilla LLM |
|
Vanilla VLM LLM engine |
|
VLM visual engine |
|
Orin NX / Orin Nano VLM visual engine |
|
EAGLE3 base engine |
Vanilla LLM parameters plus |
EAGLE3 draft engine |
Vanilla LLM parameters plus |
MTP base engine |
Vanilla LLM parameters plus |
MTP draft engine |
Vanilla LLM parameters plus |
DFlash base engine |
Vanilla LLM parameters plus |
DFlash draft engine |
Vanilla LLM parameters plus |
JetSpec tree base engine |
Vanilla LLM parameters plus |
JetSpec draft engine |
Vanilla LLM parameters plus |
Use the batch size shown in the benchmark row. Thor and Jetson AGX Orin rows may
use batch 1 or 8; Jetson Orin NX and Orin Nano rows are generally batch 1.
For INT4 runs on Orin, follow the export docs but use externalized INT4 weights:
--externalize-weights int4_ffn for dense checkpoints and
--externalize-weights int4_ffn int4_moe for MoE checkpoints.
For speculative decoding, follow the exact export layouts in
Speculative Decoding: EAGLE3 uses a base
and draft export, MTP uses the MTP base and mtp_draft export, and DFlash uses
the paired DFlash base and draft export. One DFlash base artifact supports both
linear (--spec-draft-top-k 1) and DDTree (--spec-draft-top-k > 1) verification,
within its built draft and verification token capacities. JetSpec uses the paired
JetSpec base and draft export with --spec-draft-top-k > 1.
3. Runtime Benchmark Specs#
Use llm_inference with the generated datasets and --dump-profile to collect
runtime prefill, generation, visual, memory, and speculative-decoding metrics in
the profile JSON. Use these common runtime settings:
Workload |
|
|---|---|
LLM runtime benchmark |
|
VLM runtime benchmark |
|
All runtime benchmarks |
|
EAGLE3 runtime |
Common settings plus |
MTP runtime |
Common settings plus |
DFlash linear runtime |
Common settings plus |
DFlash DDTree runtime |
Follow the DFlash guide; use |
JetSpec tree runtime |
Common settings plus |
For Qwen3.5 DFlash inputs, set "enable_thinking": true; for Qwen3 DFlash and
Qwen3 JetSpec inputs, set "enable_thinking": false. These settings match the
paired HuggingFace generation behavior used for speculative decoding
validation.
For repeatable runtime benchmarks, llm_bench uses the same deployment options
and handleRequest() path as llm_inference. Supply the engine directory and
checkpoint paths required by that deployment, including --draftCheckpointDir
for an independent draft. Configure speculative decoding with the same
--specDecode, drafting, and verification options as inference.
llm_bench --engine-dir <engine-dir> --checkpoint-dir <checkpoint-dir> \
--mode fixed_round --input-len 2048 --batch-size 1 \
--warmup 3 --iterations 10 --output-dir <new-report-dir>
llm_bench --engine-dir <engine-dir> --checkpoint-dir <checkpoint-dir> \
--mode generate --input-len 2048 --osl 128 --batch-size 1 \
--warmup 3 --iterations 10 --output-dir <another-new-report-dir>
fixed_round repeats prefill and exactly one complete decode round; generate
repeats complete requests up to the output-token budget. Both ignore EOS by
default and disable cross-request reuse, including vision/audio encoder
embedding caches. Each repeated media request executes its encoder; the bench
sets --encoderCacheBudgetBytes to 0 and rejects explicit nonzero values.
llm_inference retains its normal encoder-cache default.
--respectEos enables natural EOS
termination, but a fixed-round request that finishes before its decode round
is rejected as an invalid sample. Use --inputFile for actual inference
request JSON, including media, instead of synthetic token IDs. Synthetic input
length is an exact token count, not a repeated-text approximation.
For a supported speculative deployment, --forceAL N sets the forced acceptance
length including one target bonus token. A complete decode round accepts a
legal prefix of N - 1 draft tokens and then emits the target token from the
verification position after that prefix. The initial token from base prefill is
separate and does not count toward AL.
Option |
Accepted draft tokens per complete round |
Target bonus |
Total AL |
|---|---|---|---|
Omitted |
Determined by natural acceptance |
Normal decoder behavior |
Variable |
|
0 |
1 |
1 |
|
1 |
1 |
2 |
|
3 |
1 |
4 |
N must be a positive integer and must fit the deployment’s acceptance capacity
and the actual proposal path. Zero, negative CLI values, and unavailable paths
are rejected. The C++ control is forcedAcceptanceLength; its default -1
disables forcing. To force K accepted draft tokens, set N to K + 1; for example,
three drafts plus the bonus require --forceAL 4.
fixed_round derives its output budget from the resolved deployment, including
any normalization of the verification window. It requires enough output and KV
capacity for prefill plus one complete round; diffusion uses its full canvas
without an additional prefill token.
In generate, the final output budget can truncate the accepted prefix before
the bonus: with --forceAL 4 and two output slots remaining, the round emits two
draft tokens and no bonus. --respectEos can also stop output early. Reports
retain these actual counts; the configured AL is not a promise that every round
emits exactly N tokens after stopping rules.
Base and draft forwards and natural verification still execute. The
forceAcceptanceKernels helpers override the acceptance outputs before the normal
KV, hidden-state and recurrent-state commit, keeping token IDs, accepted indices
and lengths consistent. They run only when forcing is enabled and their cost is
included in the measured runtime stages and request time. They do not implement
a separate model execution path.
These are controlled workload measurements, not natural acceptance rates or
model speedup claims. Omit --forceAL to measure natural acceptance. The report’s
forced_acceptance_length records the requested N (-1 for natural acceptance);
in forced mode, per-round accepted_drafts, bonus_tokens and output_tokens
describe the actual emitted tokens. Draft-token counters exclude the bonus in
both modes; natural acceptance retains the estimated accounting described below.
The new output directory contains report.json (schema version 2) and CSV
views of requests and invocations. source_revision and source_dirty_at_build
identify the source tree when the benchmark executable was last built, including
incremental builds without a new CMake configure. Runtime timing storage is
preallocated for ordinary requests; longer-lived profiling can grow the pool
without an explicit stream or event synchronization. Any such allocation cost
is part of the instrumented execution. Raw samples distinguish request wall time,
production stage GPU time, and semantic base/draft forward GPU time recorded by
the runtime or decoder; do not add these nested times together. Each forward
covers the caller’s prepare and execute calls; sampling and subsequent cache
bookkeeping remain in the enclosing stage. Forward samples
use forward/base or forward/draft, retain their parent stage and actual
shapes, and carry a round-local forward_index; stage samples use -1.
query_tokens counts physical forward rows, including padding. An empty
past_lengths means no fresh host-side lengths are available, as in GPU-prepared
speculative steps. Profiling does not add a device readback to obtain them. Both
sample kinds share the invocation_gpu_ms summary. The report does not expose
engine filenames as independent timing categories. An observed graph_capture
stage marks the request non-steady, preserving its wall time separately.
Low-level CUDA graph replay and fallback are inspected with nsys, not inferred
from these samples. iterations always means full-request repetitions after
warmup. Request throughput aggregates all batch entries;
component timings are milliseconds per invocation. Acceptance samples mark
provenance_exact: forced-K accepted, bonus and output counts are exact, while
natural counts retain the runtime’s existing estimate (including its
final-budget/bonus convention). proposed_drafts_exact is false for tree
topologies built by DDTree (MTP, Gemma4 MTP, DFlash and DSpark trees), whose
proposed_drafts is the verify-tree capacity rather than the valid proposal
nodes.
Input/output FNV-1a64 fingerprints are reproducibility diagnostics, not
cryptographic integrity guarantees. Input-file fingerprints do not cover the
contents of referenced media files.
Use --noGpuTiming for wall-time/NVTX measurements and nsys for kernel
analysis; acceptance samples are still recorded without CUDA-event timing. TensorRT layer profiling (--profile, --extractLayerInfo) and old
standalone component modes have been removed. llm_inference --dumpProfile
continues to provide the shared runtime profiling information. Historical
results below describe their original SDK’s component benchmark and must not
be compared directly with the new request/stage timing boundaries.
v0.10.0 Results#
SDK Version: TensorRT Edge-LLM 0.10.0 | Jetson: JetPack 7.2, CUDA 13.2, TensorRT 10.16 | DGX Spark: CUDA 13.0, TensorRT 10.16
Decode throughput: Runtime
Decode (tok/s)reports generated tokens per second for vanilla decoding and overall accepted-token throughput for speculative decoding.llm_benchBS=8 decode throughput is reported as aggregate batch throughput.
v0.10.0 llm_bench Component Performance#
These rows report fixed-length llm_bench prefill and decode measurements at the batch sizes shown. Batch-8 decode is shown as aggregate throughput (per-sequence tok/s x batch size).
Platform |
Model |
Kind |
Mode |
Precision |
Batch |
Prefill Seq Len |
Prefill E2E (ms) |
Prefill (tok/s) |
Decode Past KV Len |
Decode (tok/s) |
|---|---|---|---|---|---|---|---|---|---|---|
Jetson AGX Thor |
gemma-4-12B-it |
VLM |
Vanilla |
FP16 / FP16 |
1 |
2,048 |
622.8 |
3,288.6 |
2,048 |
10.1 |
Jetson AGX Thor |
gemma-4-12B-it |
VLM |
Vanilla |
FP16 / FP16 |
8 |
2,048 |
7,291.6 |
280.9 |
2,048 |
74.4 |
Jetson AGX Thor |
gemma-4-31B-it |
VLM |
Vanilla |
FP8 / FP16 |
1 |
2,048 |
1,036.4 |
1,976.1 |
2,048 |
7.4 |
Jetson AGX Thor |
gemma-4-31B-it |
VLM |
Vanilla |
FP8 / FP16 |
8 |
2,048 |
9,171.5 |
223.3 |
2,048 |
48.0 |
Jetson AGX Thor |
gemma-4-31B-it |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
2,899.5 |
706.3 |
2,048 |
12.8 |
Jetson AGX Thor |
gemma-4-31B-it |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,048 |
36,592.2 |
56.0 |
2,048 |
74.4 |
Jetson AGX Thor |
nvidia-Gemma-4-26B-A4B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
187.9 |
10,901.7 |
2,048 |
41.8 |
Jetson AGX Thor |
nvidia-Gemma-4-26B-A4B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
1,718.1 |
1,192.0 |
2,048 |
193.6 |
Jetson AGX Thor |
nvidia-Gemma-4-31B-IT |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
786.9 |
2,602.5 |
2,048 |
7.7 |
Jetson AGX Thor |
nvidia-Gemma-4-31B-IT |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
10,663.3 |
192.1 |
2,048 |
54.4 |
Jetson AGX Thor |
NVIDIA-Nemotron-3-Nano-30B-A3B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
168.7 |
12,138.6 |
2,048 |
76.2 |
Jetson AGX Thor |
NVIDIA-Nemotron-3-Nano-30B-A3B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
1,033.3 |
1,982.0 |
2,048 |
275.2 |
Jetson AGX Thor |
NVIDIA-Nemotron-3-Nano-4B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
1,228.4 |
1,667.3 |
2,048 |
66.8 |
Jetson AGX Thor |
NVIDIA-Nemotron-3-Nano-4B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
5,724.6 |
357.8 |
2,048 |
423.2 |
Jetson AGX Thor |
NVIDIA-Nemotron-3.5-Lightning-30B-A3B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
376.5 |
5,439.0 |
2,048 |
80.7 |
Jetson AGX Thor |
NVIDIA-Nemotron-3.5-Lightning-30B-A3B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
2,885.1 |
709.9 |
2,048 |
282.4 |
Jetson AGX Thor |
Qwen2.5-VL-7B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
83.1 |
24,637.4 |
2,048 |
59.2 |
Jetson AGX Thor |
Qwen2.5-VL-7B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
961.9 |
2,129.2 |
2,048 |
382.4 |
Jetson AGX Thor |
Qwen3-0.6B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
54.8 |
37,350.2 |
2,048 |
253.1 |
Jetson AGX Thor |
Qwen3-0.6B |
LLM |
Vanilla |
INT4 AWQ |
8 |
2,048 |
658.4 |
3,110.6 |
2,048 |
718.4 |
Jetson AGX Thor |
Qwen3-0.6B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
17.9 |
114,242.2 |
2,048 |
244.3 |
Jetson AGX Thor |
Qwen3-0.6B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
209.3 |
9,785.5 |
2,048 |
740.0 |
Jetson AGX Thor |
Qwen3-1.7B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
145.4 |
14,082.4 |
2,048 |
135.1 |
Jetson AGX Thor |
Qwen3-1.7B |
LLM |
Vanilla |
INT4 AWQ |
8 |
2,048 |
1,777.8 |
1,152.0 |
2,048 |
547.2 |
Jetson AGX Thor |
Qwen3-1.7B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
27.7 |
73,858.3 |
2,048 |
135.5 |
Jetson AGX Thor |
Qwen3-1.7B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
314.4 |
6,514.6 |
2,048 |
532.8 |
Jetson AGX Thor |
Qwen3-30B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
1 |
2,048 |
490.6 |
4,174.4 |
2,048 |
76.7 |
Jetson AGX Thor |
Qwen3-30B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
8 |
2,048 |
4,264.1 |
480.3 |
2,048 |
233.6 |
Jetson AGX Thor |
Qwen3-30B-A3B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
137.7 |
14,869.9 |
2,048 |
84.1 |
Jetson AGX Thor |
Qwen3-30B-A3B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
964.5 |
2,123.4 |
2,048 |
251.2 |
Jetson AGX Thor |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
364.6 |
5,617.3 |
2,048 |
76.5 |
Jetson AGX Thor |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
INT4 AWQ |
8 |
2,048 |
4,671.1 |
438.4 |
2,048 |
350.4 |
Jetson AGX Thor |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
62.0 |
33,029.8 |
2,048 |
73.4 |
Jetson AGX Thor |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
746.3 |
2,744.4 |
2,048 |
356.0 |
Jetson AGX Thor |
Qwen3-8B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
663.4 |
3,087.2 |
2,048 |
46.4 |
Jetson AGX Thor |
Qwen3-8B |
LLM |
Vanilla |
INT4 AWQ |
8 |
2,048 |
8,266.6 |
247.7 |
2,048 |
250.4 |
Jetson AGX Thor |
Qwen3-8B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
104.9 |
19,516.9 |
2,048 |
44.6 |
Jetson AGX Thor |
Qwen3-8B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
1,144.7 |
1,789.1 |
2,048 |
254.4 |
Jetson AGX Thor |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
145.3 |
14,091.7 |
2,048 |
135.3 |
Jetson AGX Thor |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,048 |
1,846.1 |
1,109.4 |
2,048 |
541.6 |
Jetson AGX Thor |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
28.0 |
73,082.5 |
2,048 |
136.0 |
Jetson AGX Thor |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
317.7 |
6,446.4 |
2,048 |
564.0 |
Jetson AGX Thor |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
364.4 |
5,620.4 |
2,048 |
76.3 |
Jetson AGX Thor |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,048 |
4,658.7 |
439.6 |
2,048 |
353.6 |
Jetson AGX Thor |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
62.2 |
32,915.8 |
2,048 |
72.9 |
Jetson AGX Thor |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
750.5 |
2,728.8 |
2,048 |
358.4 |
Jetson AGX Thor |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
662.2 |
3,092.9 |
2,048 |
46.2 |
Jetson AGX Thor |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,048 |
7,897.8 |
259.3 |
2,048 |
249.6 |
Jetson AGX Thor |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
104.5 |
19,604.5 |
2,048 |
44.4 |
Jetson AGX Thor |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
1,149.5 |
1,781.7 |
2,048 |
254.4 |
Jetson AGX Thor |
Qwen3.5-0.8B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
72.5 |
28,237.7 |
2,048 |
232.5 |
Jetson AGX Thor |
Qwen3.5-0.8B |
LLM |
Vanilla |
INT4 AWQ |
8 |
2,048 |
821.0 |
2,494.5 |
2,048 |
1,183.2 |
Jetson AGX Thor |
Qwen3.5-0.8B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
32.7 |
62,644.7 |
2,048 |
229.6 |
Jetson AGX Thor |
Qwen3.5-0.8B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
327.0 |
6,263.0 |
2,048 |
1,212.8 |
Jetson AGX Thor |
Qwen3.5-0.8B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
72.4 |
28,286.9 |
2,048 |
232.6 |
Jetson AGX Thor |
Qwen3.5-0.8B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,048 |
800.9 |
2,557.2 |
2,048 |
1,172.8 |
Jetson AGX Thor |
Qwen3.5-0.8B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
32.7 |
62,606.0 |
2,048 |
229.6 |
Jetson AGX Thor |
Qwen3.5-0.8B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
327.1 |
6,261.2 |
2,048 |
1,217.6 |
Jetson AGX Thor |
Qwen3.5-27B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
2,353.1 |
870.3 |
2,048 |
15.2 |
Jetson AGX Thor |
Qwen3.5-27B |
LLM |
Vanilla |
INT4 AWQ |
8 |
2,048 |
30,171.7 |
67.9 |
2,048 |
100.8 |
Jetson AGX Thor |
Qwen3.5-27B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
448.7 |
4,563.9 |
2,048 |
14.7 |
Jetson AGX Thor |
Qwen3.5-27B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
4,234.7 |
483.6 |
2,048 |
95.2 |
Jetson AGX Thor |
Qwen3.5-27B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
2,311.9 |
885.9 |
2,048 |
15.0 |
Jetson AGX Thor |
Qwen3.5-27B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,048 |
29,907.2 |
68.5 |
2,048 |
101.6 |
Jetson AGX Thor |
Qwen3.5-27B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
447.3 |
4,578.5 |
2,048 |
14.5 |
Jetson AGX Thor |
Qwen3.5-27B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
4,213.9 |
486.0 |
2,048 |
95.2 |
Jetson AGX Thor |
Qwen3.5-2B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
152.4 |
13,440.9 |
2,048 |
122.5 |
Jetson AGX Thor |
Qwen3.5-2B |
LLM |
Vanilla |
INT4 AWQ |
8 |
2,048 |
1,858.5 |
1,101.9 |
2,048 |
738.4 |
Jetson AGX Thor |
Qwen3.5-2B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
42.9 |
47,716.2 |
2,048 |
122.4 |
Jetson AGX Thor |
Qwen3.5-2B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
423.5 |
4,835.8 |
2,048 |
752.8 |
Jetson AGX Thor |
Qwen3.5-2B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
151.7 |
13,499.3 |
2,048 |
122.8 |
Jetson AGX Thor |
Qwen3.5-2B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,048 |
1,869.2 |
1,095.7 |
2,048 |
736.8 |
Jetson AGX Thor |
Qwen3.5-2B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
43.3 |
47,327.5 |
2,048 |
122.5 |
Jetson AGX Thor |
Qwen3.5-2B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
419.4 |
4,883.7 |
2,048 |
772.8 |
Jetson AGX Thor |
Qwen3.5-35B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
1 |
2,048 |
358.0 |
5,721.4 |
2,048 |
47.3 |
Jetson AGX Thor |
Qwen3.5-35B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
8 |
2,048 |
2,898.8 |
706.5 |
2,048 |
212.0 |
Jetson AGX Thor |
Qwen3.5-35B-A3B |
VLM |
Vanilla |
INT4 GPTQ / FP16 |
1 |
2,048 |
360.0 |
5,688.9 |
2,048 |
46.7 |
Jetson AGX Thor |
Qwen3.5-35B-A3B |
VLM |
Vanilla |
INT4 GPTQ / FP16 |
8 |
2,048 |
2,891.7 |
708.2 |
2,048 |
212.8 |
Jetson AGX Thor |
Qwen3.5-4B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
370.2 |
5,531.9 |
2,048 |
69.8 |
Jetson AGX Thor |
Qwen3.5-4B |
LLM |
Vanilla |
INT4 AWQ |
8 |
2,048 |
4,771.8 |
429.2 |
2,048 |
384.8 |
Jetson AGX Thor |
Qwen3.5-4B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
96.1 |
21,316.3 |
2,048 |
67.9 |
Jetson AGX Thor |
Qwen3.5-4B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
943.0 |
2,171.8 |
2,048 |
397.6 |
Jetson AGX Thor |
Qwen3.5-4B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
371.4 |
5,515.0 |
2,048 |
69.5 |
Jetson AGX Thor |
Qwen3.5-4B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,048 |
4,693.0 |
436.4 |
2,048 |
388.0 |
Jetson AGX Thor |
Qwen3.5-4B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
95.2 |
21,507.3 |
2,048 |
67.9 |
Jetson AGX Thor |
Qwen3.5-4B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
941.3 |
2,175.7 |
2,048 |
392.0 |
Jetson AGX Thor |
Qwen3.5-9B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
677.9 |
3,020.9 |
2,048 |
41.3 |
Jetson AGX Thor |
Qwen3.5-9B |
LLM |
Vanilla |
INT4 AWQ |
8 |
2,048 |
7,979.9 |
256.6 |
2,048 |
260.0 |
Jetson AGX Thor |
Qwen3.5-9B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
134.7 |
15,205.2 |
2,048 |
40.1 |
Jetson AGX Thor |
Qwen3.5-9B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
1,390.4 |
1,473.0 |
2,048 |
259.2 |
Jetson AGX Thor |
Qwen3.5-9B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
690.5 |
2,966.2 |
2,048 |
41.3 |
Jetson AGX Thor |
Qwen3.5-9B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,048 |
8,068.1 |
253.8 |
2,048 |
256.8 |
Jetson AGX Thor |
Qwen3.5-9B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
135.3 |
15,139.5 |
2,048 |
40.1 |
Jetson AGX Thor |
Qwen3.5-9B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
1,365.3 |
1,500.1 |
2,048 |
259.2 |
Jetson AGX Thor |
Qwen3.6-35B-A3B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
187.0 |
10,950.3 |
2,048 |
84.5 |
Jetson AGX Thor |
Qwen3.6-35B-A3B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
1,194.2 |
1,715.0 |
2,048 |
264.8 |
Jetson AGX Thor |
Qwen3.6-35B-A3B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
188.5 |
10,865.8 |
2,048 |
85.0 |
Jetson AGX Thor |
Qwen3.6-35B-A3B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
1,200.8 |
1,705.5 |
2,048 |
264.0 |
Jetson AGX Orin (64GB) |
Qwen3-0.6B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
118.5 |
17,277.3 |
2,048 |
162.5 |
Jetson AGX Orin (64GB) |
Qwen3-1.7B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
308.3 |
6,643.9 |
2,048 |
89.5 |
Jetson AGX Orin (64GB) |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
761.9 |
2,687.9 |
2,048 |
50.7 |
Jetson AGX Orin (64GB) |
Qwen3-8B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
1,384.1 |
1,479.7 |
2,048 |
30.6 |
Jetson AGX Orin (64GB) |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
298.4 |
6,863.4 |
2,048 |
88.7 |
Jetson AGX Orin (64GB) |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
757.7 |
2,702.9 |
2,048 |
50.8 |
Jetson AGX Orin (64GB) |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
1,388.1 |
1,475.5 |
2,048 |
30.6 |
Jetson AGX Orin (64GB) |
Qwen3.5-0.8B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
559.8 |
3,658.6 |
2,048 |
153.7 |
Jetson AGX Orin (64GB) |
Qwen3.5-0.8B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
559.9 |
3,657.9 |
2,048 |
153.3 |
Jetson AGX Orin (64GB) |
Qwen3.5-27B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
6,352.6 |
322.4 |
2,048 |
10.5 |
Jetson AGX Orin (64GB) |
Qwen3.5-27B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
6,319.5 |
324.1 |
2,048 |
10.5 |
Jetson AGX Orin (64GB) |
Qwen3.5-2B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
730.9 |
2,802.2 |
2,048 |
81.8 |
Jetson AGX Orin (64GB) |
Qwen3.5-2B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
721.4 |
2,839.0 |
2,048 |
81.7 |
Jetson AGX Orin (64GB) |
Qwen3.5-4B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
1,374.4 |
1,490.1 |
2,048 |
46.2 |
Jetson AGX Orin (64GB) |
Qwen3.5-4B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
1,369.8 |
1,495.1 |
2,048 |
46.1 |
Jetson AGX Orin (64GB) |
Qwen3.5-9B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
2,016.5 |
1,015.6 |
2,048 |
27.6 |
Jetson AGX Orin (64GB) |
Qwen3.5-9B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
2,140.5 |
956.8 |
2,048 |
27.5 |
Jetson Orin NX (16GB) |
Qwen3-0.6B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
268.8 |
7,618.8 |
2,048 |
98.9 |
Jetson Orin NX (16GB) |
Qwen3-1.7B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
674.0 |
3,038.8 |
2,048 |
52.2 |
Jetson Orin NX (16GB) |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
1,880.5 |
1,089.1 |
2,048 |
29.2 |
Jetson Orin NX (16GB) |
Qwen3-8B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
2,944.1 |
695.6 |
2,048 |
17.5 |
Jetson Orin NX (16GB) |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
760.3 |
2,693.8 |
2,048 |
52.6 |
Jetson Orin NX (16GB) |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
1,740.8 |
1,176.4 |
2,048 |
29.4 |
Jetson Orin NX (16GB) |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
3,106.2 |
659.3 |
2,048 |
17.4 |
Jetson Orin NX (16GB) |
Qwen3.5-0.8B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
840.8 |
2,435.8 |
2,048 |
92.8 |
Jetson Orin NX (16GB) |
Qwen3.5-0.8B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
787.2 |
2,601.6 |
2,048 |
94.1 |
Jetson Orin NX (16GB) |
Qwen3.5-2B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
1,160.8 |
1,764.2 |
2,048 |
47.0 |
Jetson Orin NX (16GB) |
Qwen3.5-2B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
1,173.4 |
1,745.3 |
2,048 |
47.3 |
Jetson Orin NX (16GB) |
Qwen3.5-4B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
2,557.7 |
800.7 |
2,048 |
26.6 |
Jetson Orin NX (16GB) |
Qwen3.5-4B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
2,775.8 |
737.8 |
2,048 |
26.8 |
Jetson Orin Nano (8GB) |
Qwen3-0.6B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
445.2 |
4,600.5 |
2,048 |
61.0 |
Jetson Orin Nano (8GB) |
Qwen3-1.7B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
1,021.0 |
2,005.9 |
2,048 |
32.3 |
Jetson Orin Nano (8GB) |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
1,017.0 |
2,013.7 |
2,048 |
32.3 |
Jetson Orin Nano (8GB) |
Qwen3.5-0.8B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
1,380.8 |
1,483.2 |
2,048 |
57.5 |
Jetson Orin Nano (8GB) |
Qwen3.5-0.8B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
1,381.5 |
1,482.4 |
2,048 |
57.4 |
Jetson Orin Nano (8GB) |
Qwen3.5-2B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
1,885.2 |
1,086.4 |
2,048 |
29.0 |
Jetson Orin Nano (8GB) |
Qwen3.5-2B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
1,885.6 |
1,086.1 |
2,048 |
29.1 |
DGX Spark (GB10) |
gemma-4-12B-it |
VLM |
Vanilla |
FP16 / FP16 |
1 |
2,048 |
835.5 |
2,451.2 |
2,048 |
9.1 |
DGX Spark (GB10) |
gemma-4-12B-it |
VLM |
Vanilla |
FP16 / FP16 |
8 |
2,048 |
5,711.0 |
358.6 |
2,048 |
68.8 |
DGX Spark (GB10) |
gemma-4-31B-it |
VLM |
Vanilla |
FP8 / FP16 |
1 |
2,048 |
1,001.9 |
2,044.0 |
2,048 |
6.2 |
DGX Spark (GB10) |
gemma-4-31B-it |
VLM |
Vanilla |
FP8 / FP16 |
8 |
2,048 |
8,025.5 |
255.2 |
2,048 |
40.0 |
DGX Spark (GB10) |
gemma-4-31B-it |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
1,683.7 |
1,216.4 |
2,048 |
11.8 |
DGX Spark (GB10) |
gemma-4-31B-it |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,048 |
41,137.9 |
49.8 |
2,048 |
68.8 |
DGX Spark (GB10) |
nvidia-Gemma-4-26B-A4B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
360.4 |
5,682.1 |
2,048 |
37.1 |
DGX Spark (GB10) |
nvidia-Gemma-4-26B-A4B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
2,536.5 |
807.4 |
2,048 |
164.0 |
DGX Spark (GB10) |
nvidia-Gemma-4-31B-IT |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
1,146.3 |
1,786.7 |
2,048 |
6.5 |
DGX Spark (GB10) |
nvidia-Gemma-4-31B-IT |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
8,194.9 |
249.9 |
2,048 |
51.2 |
DGX Spark (GB10) |
NVIDIA-Nemotron-3-Nano-30B-A3B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
313.3 |
6,536.3 |
2,048 |
71.6 |
DGX Spark (GB10) |
NVIDIA-Nemotron-3-Nano-30B-A3B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
2,174.9 |
941.7 |
2,048 |
258.4 |
DGX Spark (GB10) |
NVIDIA-Nemotron-3-Nano-4B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
666.6 |
3,072.5 |
2,048 |
61.4 |
DGX Spark (GB10) |
NVIDIA-Nemotron-3-Nano-4B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
2,894.6 |
707.5 |
2,048 |
400.0 |
DGX Spark (GB10) |
NVIDIA-Nemotron-3.5-Lightning-30B-A3B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
280.0 |
7,314.8 |
2,048 |
74.3 |
DGX Spark (GB10) |
NVIDIA-Nemotron-3.5-Lightning-30B-A3B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
1,977.7 |
1,035.6 |
2,048 |
268.0 |
DGX Spark (GB10) |
Qwen2.5-VL-7B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
167.8 |
12,202.9 |
2,048 |
51.9 |
DGX Spark (GB10) |
Qwen2.5-VL-7B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
1,271.3 |
1,610.9 |
2,048 |
350.4 |
DGX Spark (GB10) |
Qwen3-0.6B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
42.2 |
48,474.3 |
2,048 |
251.4 |
DGX Spark (GB10) |
Qwen3-0.6B |
LLM |
Vanilla |
INT4 AWQ |
8 |
2,048 |
747.5 |
2,739.9 |
2,048 |
703.2 |
DGX Spark (GB10) |
Qwen3-0.6B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
29.9 |
68,558.9 |
2,048 |
214.0 |
DGX Spark (GB10) |
Qwen3-0.6B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
265.5 |
7,714.8 |
2,048 |
674.4 |
DGX Spark (GB10) |
Qwen3-1.7B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
98.7 |
20,750.1 |
2,048 |
133.9 |
DGX Spark (GB10) |
Qwen3-1.7B |
LLM |
Vanilla |
INT4 AWQ |
8 |
2,048 |
2,069.0 |
989.9 |
2,048 |
525.6 |
DGX Spark (GB10) |
Qwen3-1.7B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
55.5 |
36,909.8 |
2,048 |
116.6 |
DGX Spark (GB10) |
Qwen3-1.7B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
454.5 |
4,505.9 |
2,048 |
520.8 |
DGX Spark (GB10) |
Qwen3-30B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
1 |
2,048 |
285.0 |
7,187.2 |
2,048 |
81.1 |
DGX Spark (GB10) |
Qwen3-30B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
8 |
2,048 |
2,949.8 |
694.3 |
2,048 |
235.2 |
DGX Spark (GB10) |
Qwen3-30B-A3B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
282.4 |
7,251.0 |
2,048 |
74.7 |
DGX Spark (GB10) |
Qwen3-30B-A3B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
2,039.9 |
1,004.0 |
2,048 |
224.0 |
DGX Spark (GB10) |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
224.4 |
9,128.4 |
2,048 |
73.9 |
DGX Spark (GB10) |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
INT4 AWQ |
8 |
2,048 |
5,254.9 |
389.7 |
2,048 |
343.2 |
DGX Spark (GB10) |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
124.2 |
16,494.0 |
2,048 |
63.1 |
DGX Spark (GB10) |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
989.4 |
2,069.9 |
2,048 |
313.6 |
DGX Spark (GB10) |
Qwen3-8B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
386.4 |
5,300.6 |
2,048 |
43.5 |
DGX Spark (GB10) |
Qwen3-8B |
LLM |
Vanilla |
INT4 AWQ |
8 |
2,048 |
10,819.0 |
189.3 |
2,048 |
240.8 |
DGX Spark (GB10) |
Qwen3-8B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
182.2 |
11,238.1 |
2,048 |
38.4 |
DGX Spark (GB10) |
Qwen3-8B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
1,381.9 |
1,482.0 |
2,048 |
227.2 |
DGX Spark (GB10) |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
98.0 |
20,899.1 |
2,048 |
133.9 |
DGX Spark (GB10) |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,048 |
2,113.6 |
969.0 |
2,048 |
522.4 |
DGX Spark (GB10) |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
54.8 |
37,386.8 |
2,048 |
119.9 |
DGX Spark (GB10) |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
475.5 |
4,307.3 |
2,048 |
512.8 |
DGX Spark (GB10) |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
226.1 |
9,056.7 |
2,048 |
73.6 |
DGX Spark (GB10) |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,048 |
5,434.3 |
376.9 |
2,048 |
340.0 |
DGX Spark (GB10) |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
122.2 |
16,753.2 |
2,048 |
61.3 |
DGX Spark (GB10) |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
956.6 |
2,141.0 |
2,048 |
321.6 |
DGX Spark (GB10) |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
380.9 |
5,377.4 |
2,048 |
42.4 |
DGX Spark (GB10) |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,048 |
9,981.0 |
205.2 |
2,048 |
252.8 |
DGX Spark (GB10) |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
185.1 |
11,067.2 |
2,048 |
39.5 |
DGX Spark (GB10) |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
1,393.8 |
1,469.4 |
2,048 |
239.2 |
DGX Spark (GB10) |
Qwen3.5-0.8B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
54.5 |
37,597.5 |
2,048 |
228.0 |
DGX Spark (GB10) |
Qwen3.5-0.8B |
LLM |
Vanilla |
INT4 AWQ |
8 |
2,048 |
902.6 |
2,269.1 |
2,048 |
1,173.6 |
DGX Spark (GB10) |
Qwen3.5-0.8B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
38.0 |
53,842.3 |
2,048 |
207.4 |
DGX Spark (GB10) |
Qwen3.5-0.8B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
364.4 |
5,620.4 |
2,048 |
1,063.2 |
DGX Spark (GB10) |
Qwen3.5-0.8B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
53.7 |
38,104.1 |
2,048 |
237.7 |
DGX Spark (GB10) |
Qwen3.5-0.8B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,048 |
883.5 |
2,318.2 |
2,048 |
1,166.4 |
DGX Spark (GB10) |
Qwen3.5-0.8B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
39.1 |
52,318.4 |
2,048 |
204.6 |
DGX Spark (GB10) |
Qwen3.5-0.8B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
352.5 |
5,809.5 |
2,048 |
1,078.4 |
DGX Spark (GB10) |
Qwen3.5-27B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
1,329.5 |
1,540.4 |
2,048 |
13.9 |
DGX Spark (GB10) |
Qwen3.5-27B |
LLM |
Vanilla |
INT4 AWQ |
8 |
2,048 |
32,867.1 |
62.3 |
2,048 |
94.4 |
DGX Spark (GB10) |
Qwen3.5-27B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
594.0 |
3,447.7 |
2,048 |
12.7 |
DGX Spark (GB10) |
Qwen3.5-27B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
4,572.0 |
447.9 |
2,048 |
83.2 |
DGX Spark (GB10) |
Qwen3.5-27B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
1,326.1 |
1,544.3 |
2,048 |
14.3 |
DGX Spark (GB10) |
Qwen3.5-27B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,048 |
32,850.6 |
62.3 |
2,048 |
94.4 |
DGX Spark (GB10) |
Qwen3.5-27B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
606.9 |
3,374.3 |
2,048 |
13.1 |
DGX Spark (GB10) |
Qwen3.5-27B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
4,550.8 |
450.0 |
2,048 |
84.8 |
DGX Spark (GB10) |
Qwen3.5-2B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
103.8 |
19,730.0 |
2,048 |
118.4 |
DGX Spark (GB10) |
Qwen3.5-2B |
LLM |
Vanilla |
INT4 AWQ |
8 |
2,048 |
2,032.1 |
1,007.8 |
2,048 |
716.8 |
DGX Spark (GB10) |
Qwen3.5-2B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
61.0 |
33,579.6 |
2,048 |
109.5 |
DGX Spark (GB10) |
Qwen3.5-2B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
529.3 |
3,869.2 |
2,048 |
674.4 |
DGX Spark (GB10) |
Qwen3.5-2B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
105.5 |
19,407.4 |
2,048 |
115.7 |
DGX Spark (GB10) |
Qwen3.5-2B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,048 |
2,152.4 |
951.5 |
2,048 |
711.2 |
DGX Spark (GB10) |
Qwen3.5-2B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
61.1 |
33,491.5 |
2,048 |
108.9 |
DGX Spark (GB10) |
Qwen3.5-2B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
519.4 |
3,943.0 |
2,048 |
690.4 |
DGX Spark (GB10) |
Qwen3.5-35B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
1 |
2,048 |
277.1 |
7,390.7 |
2,048 |
47.3 |
DGX Spark (GB10) |
Qwen3.5-35B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
8 |
2,048 |
1,918.1 |
1,067.7 |
2,048 |
198.4 |
DGX Spark (GB10) |
Qwen3.5-35B-A3B |
VLM |
Vanilla |
INT4 GPTQ / FP16 |
1 |
2,048 |
277.6 |
7,376.6 |
2,048 |
42.7 |
DGX Spark (GB10) |
Qwen3.5-35B-A3B |
VLM |
Vanilla |
INT4 GPTQ / FP16 |
8 |
2,048 |
1,914.0 |
1,070.0 |
2,048 |
201.6 |
DGX Spark (GB10) |
Qwen3.5-4B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
238.4 |
8,590.6 |
2,048 |
65.9 |
DGX Spark (GB10) |
Qwen3.5-4B |
LLM |
Vanilla |
INT4 AWQ |
8 |
2,048 |
5,263.1 |
389.1 |
2,048 |
367.2 |
DGX Spark (GB10) |
Qwen3.5-4B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
130.9 |
15,641.8 |
2,048 |
60.7 |
DGX Spark (GB10) |
Qwen3.5-4B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
1,050.9 |
1,948.7 |
2,048 |
345.6 |
DGX Spark (GB10) |
Qwen3.5-4B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
236.7 |
8,651.6 |
2,048 |
66.3 |
DGX Spark (GB10) |
Qwen3.5-4B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,048 |
5,407.9 |
378.7 |
2,048 |
368.0 |
DGX Spark (GB10) |
Qwen3.5-4B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
130.1 |
15,743.5 |
2,048 |
61.8 |
DGX Spark (GB10) |
Qwen3.5-4B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
1,045.0 |
1,959.8 |
2,048 |
341.6 |
DGX Spark (GB10) |
Qwen3.5-9B |
LLM |
Vanilla |
INT4 AWQ |
1 |
2,048 |
402.4 |
5,089.9 |
2,048 |
38.0 |
DGX Spark (GB10) |
Qwen3.5-9B |
LLM |
Vanilla |
INT4 AWQ |
8 |
2,048 |
10,605.7 |
193.1 |
2,048 |
245.6 |
DGX Spark (GB10) |
Qwen3.5-9B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
195.6 |
10,468.2 |
2,048 |
35.4 |
DGX Spark (GB10) |
Qwen3.5-9B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
1,505.4 |
1,360.4 |
2,048 |
228.0 |
DGX Spark (GB10) |
Qwen3.5-9B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
2,048 |
402.8 |
5,084.7 |
2,048 |
35.5 |
DGX Spark (GB10) |
Qwen3.5-9B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,048 |
12,123.8 |
168.9 |
2,048 |
247.2 |
DGX Spark (GB10) |
Qwen3.5-9B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
197.9 |
10,347.2 |
2,048 |
34.0 |
DGX Spark (GB10) |
Qwen3.5-9B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
1,506.5 |
1,359.4 |
2,048 |
232.0 |
DGX Spark (GB10) |
Qwen3.6-35B-A3B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
273.9 |
7,477.0 |
2,048 |
67.1 |
DGX Spark (GB10) |
Qwen3.6-35B-A3B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
1,852.3 |
1,105.7 |
2,048 |
220.0 |
DGX Spark (GB10) |
Qwen3.6-35B-A3B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
270.5 |
7,570.2 |
2,048 |
66.5 |
DGX Spark (GB10) |
Qwen3.6-35B-A3B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
1,840.3 |
1,112.8 |
2,048 |
220.8 |
v0.10.0 Runtime Performance Dashboard#
Every row contains end-to-end decode throughput: generated-token throughput for vanilla decoding and accepted-token throughput for speculative decoding.
v0.10.0 Jetson AGX Thor#
Model |
Kind |
Mode |
Precision |
Batch |
Prefill Seq Len |
Prefill Time (ms) |
Prefill (tok/s) |
ViT Time (ms) |
ViT Tok/Run |
ViT (tok/s) |
Decode (tok/s) |
Accept Rate |
GPU Mem (MB) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
gemma-4-12B-it |
VLM |
MTP |
FP16 / FP16 / FP16 |
1 |
292 |
133.0 |
2,198.7 |
1.1 |
263 |
239,342.2 |
25.4 |
2.80 |
2,229 |
gemma-4-12B-it |
VLM |
MTP |
FP16 / FP16 / FP16 |
8 |
2,089 |
617.8 |
3,380.8 |
6.8 |
1,882 |
277,517.9 |
131.3 |
2.79 |
2,242 |
gemma-4-12B-it |
VLM |
Vanilla |
FP16 / FP16 |
1 |
292 |
146.8 |
1,992.3 |
1.1 |
264 |
240,083.8 |
10.2 |
- |
2,240 |
gemma-4-12B-it |
VLM |
Vanilla |
FP16 / FP16 |
8 |
2,089 |
654.2 |
3,192.9 |
6.8 |
1,882 |
277,109.2 |
37.3 |
- |
2,254 |
gemma-4-31B-it |
VLM |
Vanilla |
FP8 / FP16 |
1 |
292 |
220.5 |
1,326.4 |
83.8 |
263 |
3,143.1 |
7.6 |
- |
3,002 |
gemma-4-31B-it |
VLM |
Vanilla |
FP8 / FP16 |
8 |
2,089 |
1,300.8 |
1,605.7 |
595.9 |
1,882 |
3,157.7 |
33.0 |
- |
3,028 |
gemma-4-31B-it |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
292 |
504.1 |
580.1 |
83.2 |
263 |
3,166.2 |
13.3 |
- |
2,997 |
gemma-4-31B-it |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,089 |
3,368.7 |
620.0 |
598.6 |
1,882 |
3,143.0 |
60.0 |
- |
3,038 |
gemma-4-E2B-it |
VLM |
MTP |
FP16 / FP16 / FP16 |
1 |
292 |
36.5 |
8,020.2 |
16.7 |
263 |
15,768.6 |
86.7 |
1.99 |
5,032 |
gemma-4-E2B-it |
VLM |
MTP |
FP16 / FP16 / FP16 |
8 |
2,089 |
124.8 |
16,735.6 |
146.2 |
1,882 |
12,872.4 |
476.9 |
1.98 |
5,008 |
gemma-4-E2B-it |
VLM |
MTP |
FP8 / FP8 / FP16 |
1 |
292 |
28.3 |
10,335.0 |
16.7 |
263 |
15,756.1 |
121.7 |
1.95 |
5,061 |
gemma-4-E2B-it |
VLM |
MTP |
FP8 / FP8 / FP16 |
8 |
2,089 |
102.2 |
20,433.9 |
146.3 |
1,882 |
12,861.7 |
613.1 |
1.97 |
5,029 |
gemma-4-E2B-it |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
292 |
25.3 |
11,544.6 |
16.8 |
263 |
15,679.2 |
142.7 |
1.89 |
5,032 |
gemma-4-E2B-it |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
2,089 |
93.8 |
22,269.3 |
146.2 |
1,882 |
12,865.4 |
681.5 |
1.89 |
5,032 |
gemma-4-E2B-it |
VLM |
Vanilla |
FP16 / FP16 |
1 |
292 |
36.6 |
7,988.9 |
16.6 |
263 |
15,823.3 |
50.3 |
- |
4,988 |
gemma-4-E2B-it |
VLM |
Vanilla |
FP16 / FP16 |
8 |
2,089 |
122.5 |
17,049.5 |
146.0 |
1,882 |
12,886.3 |
327.7 |
- |
4,974 |
gemma-4-E2B-it |
VLM |
Vanilla |
FP8 / FP16 |
1 |
292 |
28.6 |
10,217.3 |
16.8 |
263 |
15,685.9 |
79.0 |
- |
5,024 |
gemma-4-E2B-it |
VLM |
Vanilla |
FP8 / FP16 |
8 |
2,089 |
97.9 |
21,332.0 |
145.5 |
1,882 |
12,934.4 |
485.1 |
- |
5,043 |
gemma-4-E2B-it |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
292 |
47.7 |
6,136.0 |
16.7 |
263 |
15,775.4 |
110.9 |
- |
4,996 |
gemma-4-E2B-it |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,089 |
226.0 |
9,240.5 |
145.8 |
1,882 |
12,904.6 |
636.0 |
- |
4,987 |
gemma-4-E2B-it |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
292 |
24.8 |
11,770.5 |
16.7 |
263 |
15,816.3 |
101.4 |
- |
5,052 |
gemma-4-E2B-it |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,089 |
94.3 |
22,157.1 |
146.3 |
1,882 |
12,862.9 |
551.5 |
- |
5,011 |
gemma-4-E4B-it |
VLM |
MTP |
FP16 / FP16 / FP16 |
1 |
292 |
59.8 |
4,887.9 |
16.9 |
263 |
15,604.5 |
45.9 |
1.95 |
5,894 |
gemma-4-E4B-it |
VLM |
MTP |
FP16 / FP16 / FP16 |
8 |
2,089 |
226.9 |
9,203.9 |
145.6 |
1,882 |
12,926.6 |
242.3 |
1.98 |
5,932 |
gemma-4-E4B-it |
VLM |
MTP |
FP8 / FP8 / FP16 |
1 |
292 |
44.4 |
6,589.3 |
16.9 |
263 |
15,589.2 |
72.9 |
1.97 |
5,938 |
gemma-4-E4B-it |
VLM |
MTP |
FP8 / FP8 / FP16 |
8 |
2,089 |
156.1 |
13,377.5 |
145.8 |
1,882 |
12,901.9 |
376.8 |
1.97 |
5,954 |
gemma-4-E4B-it |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
292 |
36.1 |
8,111.1 |
16.9 |
263 |
15,573.9 |
96.4 |
1.99 |
5,926 |
gemma-4-E4B-it |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
2,089 |
140.2 |
14,894.4 |
146.1 |
1,882 |
12,875.0 |
469.1 |
2.03 |
5,926 |
gemma-4-E4B-it |
VLM |
Vanilla |
FP16 / FP16 |
1 |
292 |
61.0 |
4,790.4 |
16.7 |
263 |
15,797.1 |
25.4 |
- |
5,877 |
gemma-4-E4B-it |
VLM |
Vanilla |
FP16 / FP16 |
8 |
2,089 |
225.6 |
9,259.9 |
145.3 |
1,882 |
12,951.3 |
147.5 |
- |
5,891 |
gemma-4-E4B-it |
VLM |
Vanilla |
FP8 / FP16 |
1 |
292 |
45.1 |
6,487.7 |
16.7 |
263 |
15,805.1 |
42.7 |
- |
5,939 |
gemma-4-E4B-it |
VLM |
Vanilla |
FP8 / FP16 |
8 |
2,089 |
157.9 |
13,228.0 |
145.5 |
1,882 |
12,931.6 |
245.5 |
- |
5,960 |
gemma-4-E4B-it |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
292 |
84.8 |
3,447.3 |
16.8 |
263 |
15,692.8 |
61.9 |
- |
5,916 |
gemma-4-E4B-it |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,089 |
456.1 |
4,579.2 |
145.2 |
1,882 |
12,960.3 |
363.6 |
- |
5,919 |
gemma-4-E4B-it |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
292 |
36.1 |
8,109.9 |
16.6 |
263 |
15,858.9 |
59.3 |
- |
5,940 |
gemma-4-E4B-it |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,089 |
138.9 |
15,042.1 |
145.4 |
1,882 |
12,941.4 |
350.0 |
- |
5,972 |
nvidia-Gemma-4-26B-A4B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
292 |
74.2 |
3,941.9 |
83.3 |
263 |
3,161.5 |
72.4 |
2.80 |
1,730 |
nvidia-Gemma-4-26B-A4B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
2,089 |
245.7 |
8,502.8 |
586.2 |
1,882 |
3,209.9 |
224.9 |
2.81 |
1,720 |
nvidia-Gemma-4-26B-A4B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
292 |
74.2 |
3,940.4 |
80.9 |
263 |
3,254.9 |
43.2 |
- |
1,725 |
nvidia-Gemma-4-26B-A4B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,089 |
245.8 |
8,496.8 |
580.4 |
1,882 |
3,241.7 |
142.5 |
- |
1,805 |
nvidia-Gemma-4-31B-IT |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
292 |
173.5 |
1,685.9 |
81.6 |
263 |
3,228.4 |
8.0 |
- |
2,995 |
nvidia-Gemma-4-31B-IT |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,089 |
979.2 |
2,133.1 |
585.4 |
1,882 |
3,214.1 |
31.7 |
- |
3,021 |
NVIDIA-Nemotron-3-Nano-30B-A3B |
LLM |
Vanilla |
NVFP4 |
1 |
66 |
75.6 |
870.8 |
- |
- |
- |
77.1 |
- |
966 |
NVIDIA-Nemotron-3-Nano-30B-A3B |
LLM |
Vanilla |
NVFP4 |
8 |
470 |
162.5 |
2,894.9 |
- |
- |
- |
225.1 |
- |
932 |
NVIDIA-Nemotron-3-Nano-4B |
LLM |
Vanilla |
NVFP4 |
1 |
66 |
39.0 |
1,686.9 |
- |
- |
- |
67.1 |
- |
1,071 |
NVIDIA-Nemotron-3-Nano-4B |
LLM |
Vanilla |
NVFP4 |
8 |
470 |
165.2 |
2,846.2 |
- |
- |
- |
371.4 |
- |
1,061 |
NVIDIA-Nemotron-3.5-Lightning-30B-A3B |
LLM |
DFlash |
NVFP4 / NVFP4 |
1 |
66 |
75.0 |
878.4 |
- |
- |
- |
58.7 |
2.22 |
945 |
NVIDIA-Nemotron-3.5-Lightning-30B-A3B |
LLM |
DFlash |
NVFP4 / NVFP4 |
8 |
470 |
237.9 |
1,977.1 |
- |
- |
- |
146.9 |
2.22 |
966 |
NVIDIA-Nemotron-3.5-Lightning-30B-A3B |
LLM |
MTP |
NVFP4 / NVFP4 |
1 |
66 |
75.3 |
874.4 |
- |
- |
- |
95.3 |
2.69 |
1,031 |
NVIDIA-Nemotron-3.5-Lightning-30B-A3B |
LLM |
MTP |
NVFP4 / NVFP4 |
8 |
470 |
236.5 |
1,988.5 |
- |
- |
- |
229.4 |
2.69 |
1,028 |
NVIDIA-Nemotron-3.5-Lightning-30B-A3B |
LLM |
Vanilla |
NVFP4 |
1 |
66 |
74.7 |
881.9 |
- |
- |
- |
81.6 |
- |
984 |
NVIDIA-Nemotron-3.5-Lightning-30B-A3B |
LLM |
Vanilla |
NVFP4 |
8 |
470 |
237.2 |
1,982.4 |
- |
- |
- |
236.0 |
- |
935 |
Qwen2.5-VL-7B-Instruct |
VLM |
EAGLE3 |
NVFP4 / NVFP4 / FP16 |
1 |
376 |
25.2 |
14,909.4 |
23.3 |
349 |
14,965.7 |
177.3 |
4.72 |
1,350 |
Qwen2.5-VL-7B-Instruct |
VLM |
EAGLE3 |
NVFP4 / NVFP4 / FP16 |
8 |
2,688 |
180.3 |
14,907.0 |
178.0 |
2,495 |
14,015.8 |
525.2 |
4.65 |
1,349 |
Qwen2.5-VL-7B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
376 |
25.0 |
15,033.9 |
23.4 |
349 |
14,928.8 |
61.0 |
- |
1,422 |
Qwen2.5-VL-7B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,688 |
179.4 |
14,984.3 |
177.4 |
2,495 |
14,062.5 |
307.9 |
- |
1,446 |
Qwen3-0.6B |
LLM |
Vanilla |
INT4 AWQ |
1 |
61 |
8.2 |
7,425.0 |
- |
- |
- |
319.7 |
- |
614 |
Qwen3-0.6B |
LLM |
Vanilla |
INT4 AWQ |
8 |
437 |
27.6 |
15,839.0 |
- |
- |
- |
1,624.8 |
- |
680 |
Qwen3-0.6B |
LLM |
Vanilla |
NVFP4 |
1 |
61 |
6.6 |
9,278.1 |
- |
- |
- |
306.4 |
- |
620 |
Qwen3-0.6B |
LLM |
Vanilla |
NVFP4 |
8 |
437 |
14.9 |
29,434.0 |
- |
- |
- |
1,680.5 |
- |
682 |
Qwen3-1.7B |
LLM |
EAGLE3 |
NVFP4 / NVFP4 |
1 |
61 |
8.7 |
7,046.1 |
- |
- |
- |
336.9 |
3.02 |
870 |
Qwen3-1.7B |
LLM |
EAGLE3 |
NVFP4 / NVFP4 |
8 |
437 |
22.1 |
19,786.5 |
- |
- |
- |
917.7 |
3.01 |
855 |
Qwen3-1.7B |
LLM |
Vanilla |
INT4 AWQ |
1 |
61 |
14.6 |
4,205.4 |
- |
- |
- |
154.1 |
- |
867 |
Qwen3-1.7B |
LLM |
Vanilla |
INT4 AWQ |
8 |
437 |
70.1 |
6,240.1 |
- |
- |
- |
903.2 |
- |
903 |
Qwen3-1.7B |
LLM |
Vanilla |
NVFP4 |
1 |
61 |
10.3 |
5,962.7 |
- |
- |
- |
153.6 |
- |
902 |
Qwen3-1.7B |
LLM |
Vanilla |
NVFP4 |
8 |
437 |
24.0 |
18,187.9 |
- |
- |
- |
862.2 |
- |
867 |
Qwen3-30B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
1 |
61 |
72.7 |
841.4 |
- |
- |
- |
84.4 |
- |
14,115 |
Qwen3-30B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
8 |
437 |
244.7 |
1,786.7 |
- |
- |
- |
261.4 |
- |
14,119 |
Qwen3-30B-A3B |
LLM |
Vanilla |
NVFP4 |
1 |
61 |
56.0 |
1,093.7 |
- |
- |
- |
89.8 |
- |
865 |
Qwen3-30B-A3B |
LLM |
Vanilla |
NVFP4 |
8 |
437 |
97.1 |
4,503.7 |
- |
- |
- |
252.3 |
- |
880 |
Qwen3-4B-Instruct-2507 |
LLM |
DFlash |
NVFP4 / NVFP4 |
1 |
57 |
18.7 |
3,056.7 |
- |
- |
- |
104.6 |
2.26 |
1,008 |
Qwen3-4B-Instruct-2507 |
LLM |
DFlash |
NVFP4 / NVFP4 |
8 |
409 |
45.0 |
9,084.2 |
- |
- |
- |
468.7 |
2.24 |
1,003 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
INT4 AWQ |
1 |
57 |
30.8 |
1,857.8 |
- |
- |
- |
83.3 |
- |
1,006 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
INT4 AWQ |
8 |
409 |
165.4 |
2,470.2 |
- |
- |
- |
515.0 |
- |
1,034 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
NVFP4 |
1 |
57 |
18.6 |
3,070.6 |
- |
- |
- |
79.8 |
- |
1,012 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
NVFP4 |
8 |
409 |
45.1 |
9,049.2 |
- |
- |
- |
513.1 |
- |
1,038 |
Qwen3-8B |
LLM |
DFlash |
NVFP4 / NVFP4 |
1 |
61 |
28.0 |
2,188.4 |
- |
- |
- |
90.0 |
3.26 |
1,466 |
Qwen3-8B |
LLM |
DFlash |
NVFP4 / NVFP4 |
8 |
437 |
65.2 |
6,702.0 |
- |
- |
- |
409.8 |
3.16 |
1,549 |
Qwen3-8B |
LLM |
EAGLE3 |
NVFP4 / NVFP4 |
1 |
61 |
24.1 |
2,539.4 |
- |
- |
- |
153.7 |
4.16 |
1,485 |
Qwen3-8B |
LLM |
EAGLE3 |
NVFP4 / NVFP4 |
8 |
437 |
60.6 |
7,214.6 |
- |
- |
- |
474.5 |
4.08 |
1,459 |
Qwen3-8B |
LLM |
Vanilla |
INT4 AWQ |
1 |
61 |
52.2 |
1,173.1 |
- |
- |
- |
49.0 |
- |
1,521 |
Qwen3-8B |
LLM |
Vanilla |
INT4 AWQ |
8 |
437 |
313.7 |
1,393.5 |
- |
- |
- |
307.9 |
- |
1,452 |
Qwen3-8B |
LLM |
Vanilla |
NVFP4 |
1 |
61 |
27.7 |
2,208.9 |
- |
- |
- |
47.3 |
- |
1,469 |
Qwen3-8B |
LLM |
Vanilla |
NVFP4 |
8 |
437 |
65.0 |
6,724.9 |
- |
- |
- |
300.3 |
- |
1,468 |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
292 |
29.3 |
9,982.1 |
11.7 |
265 |
22,706.7 |
151.1 |
- |
916 |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,089 |
176.4 |
11,841.7 |
74.8 |
1,896 |
25,331.1 |
831.8 |
- |
920 |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
292 |
12.7 |
23,095.3 |
11.4 |
265 |
23,282.7 |
151.9 |
- |
939 |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,089 |
50.5 |
41,358.4 |
75.0 |
1,896 |
25,290.1 |
779.0 |
- |
916 |
Qwen3-VL-4B-Instruct |
VLM |
EAGLE3 |
NVFP4 / NVFP4 / FP16 |
1 |
292 |
19.8 |
14,796.4 |
11.8 |
265 |
22,506.9 |
254.0 |
4.56 |
1,069 |
Qwen3-VL-4B-Instruct |
VLM |
EAGLE3 |
NVFP4 / NVFP4 / FP16 |
8 |
2,089 |
96.3 |
21,688.9 |
75.7 |
1,896 |
25,063.7 |
643.7 |
4.49 |
1,065 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
292 |
64.5 |
4,535.6 |
11.8 |
265 |
22,592.3 |
82.8 |
- |
1,069 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,089 |
448.4 |
4,658.5 |
75.8 |
1,896 |
25,013.2 |
492.2 |
- |
1,087 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
292 |
22.3 |
13,112.6 |
11.6 |
265 |
22,946.2 |
79.2 |
- |
1,059 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,089 |
98.4 |
21,226.0 |
75.5 |
1,896 |
25,100.7 |
374.4 |
- |
1,070 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
292 |
115.1 |
2,541.4 |
15.7 |
266 |
16,956.4 |
48.5 |
- |
1,550 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,089 |
874.5 |
2,388.7 |
109.1 |
1,896 |
17,378.5 |
297.0 |
- |
1,507 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
292 |
32.1 |
9,109.7 |
15.7 |
266 |
16,880.5 |
47.1 |
- |
1,510 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,089 |
168.3 |
12,410.5 |
108.5 |
1,896 |
17,472.3 |
244.2 |
- |
1,495 |
Qwen3.5-0.8B |
LLM |
Vanilla |
INT4 AWQ |
1 |
62 |
11.4 |
5,414.9 |
- |
- |
- |
240.0 |
- |
755 |
Qwen3.5-0.8B |
LLM |
Vanilla |
INT4 AWQ |
8 |
441 |
40.8 |
10,825.3 |
- |
- |
- |
1,087.8 |
- |
765 |
Qwen3.5-0.8B |
LLM |
Vanilla |
NVFP4 |
1 |
62 |
10.5 |
5,867.1 |
- |
- |
- |
236.0 |
- |
751 |
Qwen3.5-0.8B |
LLM |
Vanilla |
NVFP4 |
8 |
441 |
27.6 |
15,971.3 |
- |
- |
- |
1,219.9 |
- |
833 |
Qwen3.5-0.8B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
296 |
12.5 |
23,747.3 |
3.8 |
266 |
69,946.2 |
325.6 |
2.06 |
812 |
Qwen3.5-0.8B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
2,118 |
53.5 |
39,546.7 |
26.5 |
1,896 |
71,618.2 |
1,047.8 |
2.07 |
863 |
Qwen3.5-0.8B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
296 |
18.7 |
15,830.9 |
3.8 |
266 |
70,242.4 |
239.8 |
- |
868 |
Qwen3.5-0.8B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,118 |
95.4 |
22,193.2 |
26.4 |
1,896 |
71,874.2 |
996.3 |
- |
802 |
Qwen3.5-0.8B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
296 |
13.2 |
22,394.0 |
3.8 |
265 |
69,231.2 |
235.5 |
- |
840 |
Qwen3.5-0.8B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,118 |
54.9 |
38,557.7 |
26.5 |
1,896 |
71,514.0 |
1,063.7 |
- |
802 |
Qwen3.5-27B |
LLM |
Vanilla |
INT4 AWQ |
1 |
62 |
165.7 |
372.6 |
- |
- |
- |
15.4 |
- |
2,692 |
Qwen3.5-27B |
LLM |
Vanilla |
INT4 AWQ |
8 |
441 |
1,125.9 |
391.8 |
- |
- |
- |
95.1 |
- |
2,705 |
Qwen3.5-27B |
LLM |
Vanilla |
NVFP4 |
1 |
62 |
93.2 |
662.3 |
- |
- |
- |
14.8 |
- |
2,714 |
Qwen3.5-27B |
LLM |
Vanilla |
NVFP4 |
8 |
441 |
284.6 |
1,550.2 |
- |
- |
- |
88.0 |
- |
2,701 |
Qwen3.5-27B |
VLM |
DFlash |
NVFP4 / NVFP4 / FP16 |
1 |
296 |
118.0 |
2,511.5 |
14.5 |
265 |
18,280.1 |
21.6 |
2.56 |
2,806 |
Qwen3.5-27B |
VLM |
DFlash |
NVFP4 / NVFP4 / FP16 |
8 |
2,118 |
675.8 |
3,133.3 |
103.7 |
1,896 |
18,281.9 |
60.4 |
2.49 |
2,759 |
Qwen3.5-27B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
296 |
110.4 |
2,684.1 |
14.4 |
265 |
18,417.1 |
35.0 |
2.82 |
2,754 |
Qwen3.5-27B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
2,118 |
664.8 |
3,185.4 |
103.2 |
1,896 |
18,364.6 |
145.9 |
2.78 |
2,823 |
Qwen3.5-27B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
296 |
394.2 |
752.0 |
14.5 |
266 |
18,323.0 |
15.4 |
- |
2,758 |
Qwen3.5-27B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,118 |
2,991.2 |
707.9 |
103.1 |
1,896 |
18,384.7 |
90.6 |
- |
2,739 |
Qwen3.5-27B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
296 |
117.5 |
2,523.5 |
14.4 |
265 |
18,382.9 |
14.8 |
- |
2,747 |
Qwen3.5-27B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,118 |
670.0 |
3,160.5 |
103.4 |
1,896 |
18,345.5 |
74.9 |
- |
2,760 |
Qwen3.5-2B |
LLM |
Vanilla |
INT4 AWQ |
1 |
62 |
18.7 |
3,308.4 |
- |
- |
- |
123.6 |
- |
1,237 |
Qwen3.5-2B |
LLM |
Vanilla |
INT4 AWQ |
8 |
441 |
79.5 |
5,546.5 |
- |
- |
- |
635.6 |
- |
1,241 |
Qwen3.5-2B |
LLM |
Vanilla |
NVFP4 |
1 |
62 |
15.1 |
4,103.1 |
- |
- |
- |
124.9 |
- |
1,279 |
Qwen3.5-2B |
LLM |
Vanilla |
NVFP4 |
8 |
441 |
36.1 |
12,210.7 |
- |
- |
- |
736.4 |
- |
1,239 |
Qwen3.5-2B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
296 |
15.8 |
18,767.1 |
10.9 |
265 |
24,286.8 |
216.0 |
2.40 |
1,284 |
Qwen3.5-2B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
2,118 |
71.7 |
29,522.3 |
72.4 |
1,896 |
26,174.8 |
620.2 |
2.40 |
1,300 |
Qwen3.5-2B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
296 |
33.8 |
8,759.0 |
10.9 |
265 |
24,447.4 |
124.5 |
- |
1,287 |
Qwen3.5-2B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,118 |
193.3 |
10,955.7 |
72.6 |
1,896 |
26,111.0 |
608.6 |
- |
1,279 |
Qwen3.5-2B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
296 |
18.2 |
16,324.2 |
10.9 |
265 |
24,307.3 |
125.4 |
- |
1,305 |
Qwen3.5-2B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,118 |
73.4 |
28,839.3 |
72.2 |
1,896 |
26,249.4 |
444.2 |
- |
1,282 |
Qwen3.5-35B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
1 |
62 |
66.5 |
929.3 |
- |
- |
- |
48.2 |
- |
15,666 |
Qwen3.5-35B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
8 |
441 |
217.5 |
2,028.0 |
- |
- |
- |
199.0 |
- |
15,652 |
Qwen3.5-35B-A3B |
VLM |
Vanilla |
INT4 GPTQ / FP16 |
1 |
296 |
116.4 |
2,547.0 |
14.3 |
265 |
18,510.3 |
47.5 |
- |
15,683 |
Qwen3.5-35B-A3B |
VLM |
Vanilla |
INT4 GPTQ / FP16 |
8 |
2,118 |
473.0 |
4,476.8 |
101.8 |
1,896 |
18,627.2 |
189.7 |
- |
15,720 |
Qwen3.5-4B |
LLM |
Vanilla |
INT4 AWQ |
1 |
62 |
36.9 |
1,675.4 |
- |
- |
- |
70.9 |
- |
1,488 |
Qwen3.5-4B |
LLM |
Vanilla |
INT4 AWQ |
8 |
441 |
193.7 |
2,277.3 |
- |
- |
- |
377.0 |
- |
1,522 |
Qwen3.5-4B |
LLM |
Vanilla |
NVFP4 |
1 |
62 |
25.2 |
2,448.5 |
- |
- |
- |
69.4 |
- |
1,480 |
Qwen3.5-4B |
LLM |
Vanilla |
NVFP4 |
8 |
441 |
72.6 |
6,078.8 |
- |
- |
- |
388.6 |
- |
1,536 |
Qwen3.5-4B |
VLM |
DFlash |
NVFP4 / NVFP4 / FP16 |
1 |
296 |
32.1 |
9,246.2 |
10.9 |
266 |
24,269.1 |
74.2 |
2.27 |
1,530 |
Qwen3.5-4B |
VLM |
DFlash |
NVFP4 / NVFP4 / FP16 |
8 |
2,118 |
151.8 |
13,951.8 |
73.1 |
1,896 |
25,936.5 |
179.1 |
2.24 |
1,595 |
Qwen3.5-4B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
296 |
28.5 |
10,400.6 |
10.9 |
265 |
24,257.5 |
126.6 |
2.46 |
1,583 |
Qwen3.5-4B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
2,118 |
148.3 |
14,279.1 |
72.8 |
1,896 |
26,061.8 |
412.1 |
2.51 |
1,527 |
Qwen3.5-4B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
296 |
72.6 |
4,085.9 |
11.0 |
265 |
24,210.6 |
71.2 |
- |
1,554 |
Qwen3.5-4B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,118 |
488.2 |
4,337.7 |
72.7 |
1,896 |
26,065.9 |
319.7 |
- |
1,529 |
Qwen3.5-4B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
296 |
31.6 |
9,379.1 |
10.9 |
265 |
24,348.3 |
69.5 |
- |
1,538 |
Qwen3.5-4B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,118 |
151.2 |
14,005.8 |
72.8 |
1,896 |
26,042.9 |
311.2 |
- |
1,523 |
Qwen3.5-9B |
LLM |
Vanilla |
INT4 AWQ |
1 |
62 |
58.8 |
1,050.1 |
- |
- |
- |
41.4 |
- |
2,208 |
Qwen3.5-9B |
LLM |
Vanilla |
INT4 AWQ |
8 |
441 |
344.0 |
1,282.5 |
- |
- |
- |
242.3 |
- |
2,259 |
Qwen3.5-9B |
LLM |
Vanilla |
NVFP4 |
1 |
62 |
36.8 |
1,677.9 |
- |
- |
- |
40.7 |
- |
2,273 |
Qwen3.5-9B |
LLM |
Vanilla |
NVFP4 |
8 |
441 |
96.1 |
4,589.4 |
- |
- |
- |
240.4 |
- |
2,231 |
Qwen3.5-9B |
VLM |
DFlash |
NVFP4 / NVFP4 / FP16 |
1 |
296 |
44.6 |
6,653.0 |
14.4 |
265 |
18,462.4 |
47.0 |
2.40 |
2,312 |
Qwen3.5-9B |
VLM |
DFlash |
NVFP4 / NVFP4 / FP16 |
8 |
2,118 |
221.3 |
9,569.1 |
101.6 |
1,896 |
18,666.0 |
139.9 |
2.35 |
2,267 |
Qwen3.5-9B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
296 |
39.3 |
7,541.7 |
14.4 |
265 |
18,404.3 |
88.0 |
2.69 |
2,310 |
Qwen3.5-9B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
2,118 |
214.1 |
9,889.8 |
102.0 |
1,896 |
18,586.0 |
346.7 |
2.70 |
2,296 |
Qwen3.5-9B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
296 |
124.4 |
2,383.0 |
14.4 |
265 |
18,424.2 |
41.9 |
- |
2,260 |
Qwen3.5-9B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,118 |
902.2 |
2,347.2 |
101.6 |
1,896 |
18,670.4 |
215.4 |
- |
2,277 |
Qwen3.5-9B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
296 |
44.0 |
6,743.0 |
14.5 |
266 |
18,348.3 |
40.9 |
- |
2,270 |
Qwen3.5-9B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,118 |
219.4 |
9,651.0 |
101.3 |
1,896 |
18,722.3 |
191.1 |
- |
2,267 |
Qwen3.6-27B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
296 |
109.6 |
2,703.8 |
14.6 |
266 |
18,212.4 |
34.4 |
2.75 |
2,764 |
Qwen3.6-27B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
2,118 |
662.3 |
3,197.3 |
103.2 |
1,896 |
18,381.6 |
135.7 |
2.76 |
2,782 |
Qwen3.6-35B-A3B |
LLM |
MTP |
NVFP4 / NVFP4 |
1 |
62 |
55.7 |
1,109.7 |
- |
- |
- |
109.3 |
2.93 |
1,327 |
Qwen3.6-35B-A3B |
LLM |
MTP |
NVFP4 / NVFP4 |
8 |
441 |
136.8 |
3,224.0 |
- |
- |
- |
293.6 |
2.94 |
1,292 |
Qwen3.6-35B-A3B |
LLM |
Vanilla |
NVFP4 |
1 |
62 |
55.7 |
1,108.0 |
- |
- |
- |
86.6 |
- |
1,286 |
Qwen3.6-35B-A3B |
LLM |
Vanilla |
NVFP4 |
8 |
441 |
136.9 |
3,221.8 |
- |
- |
- |
252.7 |
- |
1,232 |
Qwen3.6-35B-A3B |
VLM |
DFlash |
NVFP4 / NVFP4 / FP16 |
1 |
296 |
96.7 |
3,066.9 |
14.4 |
265 |
18,450.1 |
60.9 |
2.37 |
1,287 |
Qwen3.6-35B-A3B |
VLM |
DFlash |
NVFP4 / NVFP4 / FP16 |
8 |
2,118 |
241.9 |
8,754.6 |
101.8 |
1,896 |
18,625.9 |
129.5 |
2.32 |
1,336 |
Qwen3.6-35B-A3B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
296 |
96.0 |
3,089.3 |
14.4 |
266 |
18,439.1 |
102.5 |
2.73 |
1,293 |
Qwen3.6-35B-A3B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
2,118 |
244.3 |
8,666.6 |
101.2 |
1,896 |
18,728.7 |
290.3 |
2.76 |
1,331 |
Qwen3.6-35B-A3B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
296 |
96.7 |
3,067.2 |
14.4 |
265 |
18,462.2 |
86.6 |
- |
1,292 |
Qwen3.6-35B-A3B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,118 |
242.4 |
8,736.4 |
101.7 |
1,896 |
18,653.9 |
268.3 |
- |
1,312 |
v0.10.0 Jetson AGX Orin (64GB)#
Model |
Kind |
Mode |
Precision |
Batch |
Prefill Seq Len |
Prefill Time (ms) |
Prefill (tok/s) |
ViT Time (ms) |
ViT Tok/Run |
ViT (tok/s) |
Decode (tok/s) |
Accept Rate |
GPU Mem (MB) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Qwen3-0.6B |
LLM |
Vanilla |
INT4 AWQ |
1 |
61 |
12.8 |
4,766.7 |
- |
- |
- |
200.5 |
- |
1,943 |
Qwen3-1.7B |
LLM |
Vanilla |
INT4 AWQ |
1 |
61 |
24.3 |
2,523.2 |
- |
- |
- |
99.8 |
- |
2,978 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
INT4 AWQ |
1 |
57 |
49.9 |
1,146.6 |
- |
- |
- |
55.3 |
- |
4,619 |
Qwen3-8B |
LLM |
Vanilla |
INT4 AWQ |
1 |
61 |
74.7 |
819.8 |
- |
- |
- |
32.2 |
- |
7,199 |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
292 |
45.7 |
6,393.8 |
36.2 |
265 |
7,333.7 |
98.9 |
- |
4,554 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
292 |
105.7 |
2,767.5 |
36.5 |
265 |
7,270.5 |
54.7 |
- |
6,076 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
292 |
178.3 |
1,640.7 |
53.0 |
265 |
5,009.5 |
32.1 |
- |
9,004 |
Qwen3.5-0.8B |
LLM |
Vanilla |
INT4 AWQ |
1 |
62 |
23.6 |
2,617.2 |
- |
- |
- |
158.3 |
- |
2,226 |
Qwen3.5-0.8B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
296 |
83.5 |
3,552.4 |
12.2 |
266 |
21,709.2 |
157.8 |
- |
2,783 |
Qwen3.5-27B |
LLM |
Vanilla |
INT4 AWQ |
1 |
62 |
291.9 |
211.6 |
- |
- |
- |
10.6 |
- |
18,652 |
Qwen3.5-27B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
296 |
843.8 |
351.3 |
49.7 |
265 |
5,344.4 |
10.6 |
- |
19,899 |
Qwen3.5-2B |
LLM |
Vanilla |
INT4 AWQ |
1 |
62 |
33.4 |
1,846.8 |
- |
- |
- |
83.3 |
- |
3,669 |
Qwen3.5-2B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
296 |
106.7 |
2,777.6 |
34.3 |
265 |
7,735.8 |
83.2 |
- |
4,751 |
Qwen3.5-4B |
LLM |
Vanilla |
INT4 AWQ |
1 |
62 |
67.6 |
913.6 |
- |
- |
- |
47.2 |
- |
5,447 |
Qwen3.5-4B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
296 |
197.5 |
1,500.8 |
34.3 |
265 |
7,735.8 |
47.3 |
- |
6,458 |
Qwen3.5-9B |
LLM |
Vanilla |
INT4 AWQ |
1 |
62 |
93.4 |
661.0 |
- |
- |
- |
27.9 |
- |
8,614 |
Qwen3.5-9B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
296 |
275.3 |
1,076.9 |
49.7 |
265 |
5,345.8 |
27.9 |
- |
9,849 |
v0.10.0 Jetson Orin NX (16GB)#
Model |
Kind |
Mode |
Precision |
Batch |
Prefill Seq Len |
Prefill Time (ms) |
Prefill (tok/s) |
ViT Time (ms) |
ViT Tok/Run |
ViT (tok/s) |
Decode (tok/s) |
Accept Rate |
GPU Mem (MB) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Qwen3-0.6B |
LLM |
Vanilla |
INT4 AWQ |
1 |
61 |
17.6 |
3,478.1 |
- |
- |
- |
124.9 |
- |
1,914 |
Qwen3-1.7B |
LLM |
Vanilla |
INT4 AWQ |
1 |
61 |
39.5 |
1,551.2 |
- |
- |
- |
59.0 |
- |
2,999 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
INT4 AWQ |
1 |
57 |
78.4 |
729.5 |
- |
- |
- |
32.2 |
- |
4,605 |
Qwen3-8B |
LLM |
Vanilla |
INT4 AWQ |
1 |
61 |
134.5 |
455.1 |
- |
- |
- |
18.4 |
- |
7,167 |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
292 |
93.5 |
3,129.2 |
77.1 |
265 |
3,444.3 |
58.5 |
- |
4,533 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
292 |
217.2 |
1,346.2 |
77.4 |
265 |
3,428.0 |
31.7 |
- |
6,023 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
292 |
400.5 |
730.3 |
110.5 |
265 |
2,402.1 |
18.4 |
- |
8,929 |
Qwen3.5-0.8B |
LLM |
Vanilla |
INT4 AWQ |
1 |
62 |
35.5 |
1,740.9 |
- |
- |
- |
95.8 |
- |
2,204 |
Qwen3.5-0.8B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
296 |
121.0 |
2,450.0 |
25.1 |
265 |
10,597.9 |
95.3 |
- |
2,744 |
Qwen3.5-2B |
LLM |
Vanilla |
INT4 AWQ |
1 |
62 |
56.2 |
1,099.5 |
- |
- |
- |
47.9 |
- |
3,657 |
Qwen3.5-2B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
296 |
170.6 |
1,737.9 |
72.6 |
265 |
3,658.7 |
48.3 |
- |
4,718 |
Qwen3.5-4B |
LLM |
Vanilla |
INT4 AWQ |
1 |
62 |
120.1 |
514.2 |
- |
- |
- |
27.0 |
- |
5,418 |
Qwen3.5-4B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
296 |
371.1 |
798.8 |
73.3 |
265 |
3,623.6 |
27.0 |
- |
6,433 |
v0.10.0 Jetson Orin Nano (8GB)#
Model |
Kind |
Mode |
Precision |
Batch |
Prefill Seq Len |
Prefill Time (ms) |
Prefill (tok/s) |
ViT Time (ms) |
ViT Tok/Run |
ViT (tok/s) |
Decode (tok/s) |
Accept Rate |
GPU Mem (MB) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Qwen3-0.6B |
LLM |
Vanilla |
INT4 AWQ |
1 |
61 |
30.7 |
1,992.7 |
- |
- |
- |
77.0 |
- |
1,917 |
Qwen3-1.7B |
LLM |
Vanilla |
INT4 AWQ |
1 |
61 |
69.8 |
876.9 |
- |
- |
- |
36.5 |
- |
2,992 |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
292 |
167.2 |
1,749.6 |
141.3 |
265 |
1,879.2 |
36.1 |
- |
4,486 |
Qwen3.5-0.8B |
LLM |
Vanilla |
INT4 AWQ |
1 |
62 |
63.2 |
977.2 |
- |
- |
- |
59.1 |
- |
2,127 |
Qwen3.5-0.8B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
296 |
215.6 |
1,375.0 |
45.4 |
265 |
5,851.3 |
59.0 |
- |
2,760 |
Qwen3.5-2B |
LLM |
Vanilla |
INT4 AWQ |
1 |
62 |
101.4 |
609.0 |
- |
- |
- |
29.6 |
- |
3,642 |
Qwen3.5-2B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
296 |
306.8 |
966.4 |
133.4 |
265 |
1,989.9 |
29.6 |
- |
4,692 |
v0.10.0 DGX Spark (GB10)#
Model |
Kind |
Mode |
Precision |
Batch |
Prefill Seq Len |
Prefill Time (ms) |
Prefill (tok/s) |
ViT Time (ms) |
ViT Tok/Run |
ViT (tok/s) |
Decode (tok/s) |
Accept Rate |
GPU Mem (MB) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
gemma-4-12B-it |
VLM |
MTP |
FP16 / FP16 / FP16 |
1 |
292 |
160.9 |
1,816.9 |
1.3 |
263 |
205,732.6 |
23.6 |
2.80 |
2,375 |
gemma-4-12B-it |
VLM |
MTP |
FP16 / FP16 / FP16 |
8 |
2,089 |
677.0 |
3,085.3 |
6.3 |
1,882 |
297,851.7 |
118.7 |
2.80 |
2,364 |
gemma-4-12B-it |
VLM |
Vanilla |
FP16 / FP16 |
1 |
292 |
183.5 |
1,593.4 |
1.2 |
263 |
226,656.3 |
9.4 |
- |
2,380 |
gemma-4-12B-it |
VLM |
Vanilla |
FP16 / FP16 |
8 |
2,089 |
704.9 |
2,963.3 |
6.2 |
1,880 |
300,845.1 |
35.9 |
- |
2,382 |
gemma-4-31B-it |
VLM |
Vanilla |
FP8 / FP16 |
1 |
292 |
241.3 |
1,211.7 |
101.8 |
263 |
2,588.2 |
6.6 |
- |
3,146 |
gemma-4-31B-it |
VLM |
Vanilla |
FP8 / FP16 |
8 |
2,089 |
997.8 |
2,093.2 |
772.2 |
1,882 |
2,436.5 |
26.6 |
- |
3,147 |
gemma-4-31B-it |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
292 |
277.6 |
1,053.5 |
105.8 |
263 |
2,489.0 |
12.7 |
- |
3,143 |
gemma-4-31B-it |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,089 |
1,783.8 |
1,170.9 |
787.9 |
1,882 |
2,388.2 |
51.0 |
- |
3,134 |
gemma-4-E2B-it |
VLM |
MTP |
FP16 / FP16 / FP16 |
1 |
292 |
38.8 |
7,535.7 |
20.6 |
263 |
12,787.0 |
87.8 |
1.98 |
5,191 |
gemma-4-E2B-it |
VLM |
MTP |
FP16 / FP16 / FP16 |
8 |
2,089 |
138.1 |
15,124.0 |
161.9 |
1,882 |
11,619.6 |
488.8 |
1.98 |
5,213 |
gemma-4-E2B-it |
VLM |
MTP |
FP8 / FP8 / FP16 |
1 |
292 |
25.4 |
11,495.6 |
20.6 |
263 |
12,757.7 |
117.6 |
1.95 |
5,134 |
gemma-4-E2B-it |
VLM |
MTP |
FP8 / FP8 / FP16 |
8 |
2,089 |
95.8 |
21,805.2 |
162.2 |
1,882 |
11,603.7 |
615.8 |
1.93 |
5,133 |
gemma-4-E2B-it |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
292 |
23.6 |
12,408.1 |
20.8 |
263 |
12,666.0 |
136.9 |
1.91 |
5,170 |
gemma-4-E2B-it |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
2,089 |
95.4 |
21,895.3 |
161.5 |
1,882 |
11,652.3 |
711.6 |
1.88 |
5,175 |
gemma-4-E2B-it |
VLM |
Vanilla |
FP16 / FP16 |
1 |
292 |
38.9 |
7,518.7 |
19.4 |
263 |
13,601.5 |
50.9 |
- |
5,220 |
gemma-4-E2B-it |
VLM |
Vanilla |
FP16 / FP16 |
8 |
2,089 |
143.0 |
14,609.1 |
161.9 |
1,882 |
11,623.1 |
322.1 |
- |
5,159 |
gemma-4-E2B-it |
VLM |
Vanilla |
FP8 / FP16 |
1 |
292 |
25.5 |
11,484.7 |
20.7 |
263 |
12,738.5 |
74.8 |
- |
5,108 |
gemma-4-E2B-it |
VLM |
Vanilla |
FP8 / FP16 |
8 |
2,089 |
96.5 |
21,638.7 |
161.5 |
1,882 |
11,650.2 |
468.2 |
- |
5,131 |
gemma-4-E2B-it |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
292 |
27.8 |
10,528.3 |
20.8 |
263 |
12,665.6 |
111.4 |
- |
5,114 |
gemma-4-E2B-it |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,089 |
161.2 |
12,954.3 |
160.6 |
1,882 |
11,716.0 |
661.1 |
- |
5,118 |
gemma-4-E2B-it |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
292 |
22.5 |
13,007.5 |
20.8 |
263 |
12,650.3 |
93.7 |
- |
5,161 |
gemma-4-E2B-it |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,089 |
94.3 |
22,143.0 |
162.1 |
1,882 |
11,605.5 |
538.0 |
- |
5,107 |
gemma-4-E4B-it |
VLM |
MTP |
FP16 / FP16 / FP16 |
1 |
292 |
69.9 |
4,184.3 |
19.6 |
263 |
13,468.1 |
42.8 |
1.98 |
6,147 |
gemma-4-E4B-it |
VLM |
MTP |
FP16 / FP16 / FP16 |
8 |
2,089 |
265.4 |
7,870.8 |
159.8 |
1,882 |
11,774.5 |
232.5 |
1.96 |
6,059 |
gemma-4-E4B-it |
VLM |
MTP |
FP8 / FP8 / FP16 |
1 |
292 |
43.8 |
6,674.8 |
21.2 |
263 |
12,418.3 |
67.9 |
1.99 |
6,047 |
gemma-4-E4B-it |
VLM |
MTP |
FP8 / FP8 / FP16 |
8 |
2,089 |
169.9 |
12,296.2 |
160.0 |
1,882 |
11,759.2 |
344.0 |
1.97 |
6,031 |
gemma-4-E4B-it |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
292 |
37.9 |
7,709.2 |
21.1 |
263 |
12,460.3 |
89.5 |
1.98 |
5,999 |
gemma-4-E4B-it |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
2,089 |
157.9 |
13,229.9 |
158.4 |
1,882 |
11,879.8 |
499.8 |
2.00 |
6,091 |
gemma-4-E4B-it |
VLM |
Vanilla |
FP16 / FP16 |
1 |
292 |
71.2 |
4,108.4 |
21.2 |
263 |
12,448.3 |
24.0 |
- |
6,134 |
gemma-4-E4B-it |
VLM |
Vanilla |
FP16 / FP16 |
8 |
2,089 |
264.0 |
7,912.7 |
162.0 |
1,882 |
11,617.7 |
146.4 |
- |
6,070 |
gemma-4-E4B-it |
VLM |
Vanilla |
FP8 / FP16 |
1 |
292 |
44.6 |
6,558.8 |
20.9 |
263 |
12,578.3 |
37.7 |
- |
6,026 |
gemma-4-E4B-it |
VLM |
Vanilla |
FP8 / FP16 |
8 |
2,089 |
169.7 |
12,311.2 |
162.3 |
1,882 |
11,594.4 |
215.0 |
- |
6,041 |
gemma-4-E4B-it |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
292 |
47.8 |
6,121.7 |
21.1 |
263 |
12,490.3 |
60.7 |
- |
6,033 |
gemma-4-E4B-it |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,089 |
337.0 |
6,197.8 |
162.5 |
1,882 |
11,581.5 |
359.0 |
- |
6,040 |
gemma-4-E4B-it |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
292 |
36.1 |
8,100.1 |
19.6 |
263 |
13,458.1 |
54.0 |
- |
6,042 |
gemma-4-E4B-it |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,089 |
152.8 |
13,669.1 |
163.7 |
1,882 |
11,493.8 |
319.6 |
- |
6,091 |
nvidia-Gemma-4-26B-A4B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
292 |
107.6 |
2,717.0 |
110.3 |
263 |
2,387.9 |
66.6 |
2.91 |
1,849 |
nvidia-Gemma-4-26B-A4B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
2,089 |
367.7 |
5,679.8 |
762.1 |
1,882 |
2,468.9 |
194.6 |
2.83 |
1,850 |
nvidia-Gemma-4-26B-A4B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
292 |
101.7 |
2,874.5 |
105.6 |
263 |
2,493.8 |
38.9 |
- |
1,854 |
nvidia-Gemma-4-26B-A4B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,089 |
368.1 |
5,674.1 |
777.8 |
1,882 |
2,419.1 |
105.0 |
- |
1,854 |
nvidia-Gemma-4-31B-IT |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
292 |
239.8 |
1,219.2 |
108.7 |
263 |
2,423.5 |
6.9 |
- |
3,149 |
nvidia-Gemma-4-31B-IT |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,089 |
1,087.2 |
1,921.1 |
767.6 |
1,882 |
2,451.1 |
31.6 |
- |
3,118 |
NVIDIA-Nemotron-3-Nano-30B-A3B |
LLM |
Vanilla |
NVFP4 |
1 |
66 |
75.9 |
867.8 |
- |
- |
- |
73.9 |
- |
1,061 |
NVIDIA-Nemotron-3-Nano-30B-A3B |
LLM |
Vanilla |
NVFP4 |
8 |
470 |
186.3 |
2,523.9 |
- |
- |
- |
216.4 |
- |
1,063 |
NVIDIA-Nemotron-3-Nano-4B |
LLM |
Vanilla |
NVFP4 |
1 |
66 |
29.3 |
2,248.9 |
- |
- |
- |
63.5 |
- |
1,185 |
NVIDIA-Nemotron-3-Nano-4B |
LLM |
Vanilla |
NVFP4 |
8 |
470 |
99.7 |
4,717.6 |
- |
- |
- |
342.9 |
- |
1,170 |
NVIDIA-Nemotron-3.5-Lightning-30B-A3B |
LLM |
DFlash |
NVFP4 / NVFP4 |
1 |
66 |
68.6 |
960.3 |
- |
- |
- |
61.5 |
2.23 |
1,052 |
NVIDIA-Nemotron-3.5-Lightning-30B-A3B |
LLM |
DFlash |
NVFP4 / NVFP4 |
8 |
470 |
164.4 |
2,860.2 |
- |
- |
- |
149.2 |
1.66 |
1,051 |
NVIDIA-Nemotron-3.5-Lightning-30B-A3B |
LLM |
MTP |
NVFP4 / NVFP4 |
1 |
66 |
68.6 |
959.7 |
- |
- |
- |
93.2 |
2.70 |
1,068 |
NVIDIA-Nemotron-3.5-Lightning-30B-A3B |
LLM |
MTP |
NVFP4 / NVFP4 |
8 |
470 |
162.0 |
2,903.6 |
- |
- |
- |
185.0 |
1.82 |
1,078 |
NVIDIA-Nemotron-3.5-Lightning-30B-A3B |
LLM |
Vanilla |
NVFP4 |
1 |
66 |
67.5 |
976.0 |
- |
- |
- |
75.6 |
- |
1,084 |
NVIDIA-Nemotron-3.5-Lightning-30B-A3B |
LLM |
Vanilla |
NVFP4 |
8 |
470 |
162.6 |
2,892.4 |
- |
- |
- |
244.9 |
- |
1,072 |
Qwen2.5-VL-7B-Instruct |
VLM |
EAGLE3 |
NVFP4 / NVFP4 / FP16 |
1 |
376 |
37.5 |
10,025.8 |
33.2 |
349 |
10,522.7 |
154.9 |
4.72 |
1,495 |
Qwen2.5-VL-7B-Instruct |
VLM |
EAGLE3 |
NVFP4 / NVFP4 / FP16 |
8 |
2,688 |
262.8 |
10,227.5 |
243.5 |
2,495 |
10,246.7 |
452.5 |
4.62 |
1,496 |
Qwen2.5-VL-7B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
376 |
37.7 |
9,970.3 |
32.9 |
349 |
10,604.7 |
55.6 |
- |
1,482 |
Qwen2.5-VL-7B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,688 |
262.0 |
10,258.2 |
243.9 |
2,495 |
10,229.1 |
294.7 |
- |
1,483 |
Qwen3-0.6B |
LLM |
Vanilla |
INT4 AWQ |
1 |
61 |
6.7 |
9,198.4 |
- |
- |
- |
319.3 |
- |
722 |
Qwen3-0.6B |
LLM |
Vanilla |
INT4 AWQ |
8 |
437 |
17.7 |
24,694.6 |
- |
- |
- |
1,788.2 |
- |
760 |
Qwen3-0.6B |
LLM |
Vanilla |
NVFP4 |
1 |
61 |
5.8 |
10,564.7 |
- |
- |
- |
283.2 |
- |
756 |
Qwen3-0.6B |
LLM |
Vanilla |
NVFP4 |
8 |
437 |
13.8 |
31,670.5 |
- |
- |
- |
1,725.8 |
- |
805 |
Qwen3-1.7B |
LLM |
EAGLE3 |
NVFP4 / NVFP4 |
1 |
61 |
8.7 |
7,037.3 |
- |
- |
- |
291.3 |
2.96 |
1,005 |
Qwen3-1.7B |
LLM |
EAGLE3 |
NVFP4 / NVFP4 |
8 |
437 |
23.3 |
18,723.3 |
- |
- |
- |
883.3 |
3.01 |
1,002 |
Qwen3-1.7B |
LLM |
Vanilla |
INT4 AWQ |
1 |
61 |
10.7 |
5,710.5 |
- |
- |
- |
152.1 |
- |
1,005 |
Qwen3-1.7B |
LLM |
Vanilla |
INT4 AWQ |
8 |
437 |
45.1 |
9,690.8 |
- |
- |
- |
939.3 |
- |
1,007 |
Qwen3-1.7B |
LLM |
Vanilla |
NVFP4 |
1 |
61 |
10.3 |
5,958.8 |
- |
- |
- |
136.5 |
- |
1,007 |
Qwen3-1.7B |
LLM |
Vanilla |
NVFP4 |
8 |
437 |
24.8 |
17,597.0 |
- |
- |
- |
902.2 |
- |
975 |
Qwen3-30B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
1 |
61 |
63.1 |
970.4 |
- |
- |
- |
85.2 |
- |
14,236 |
Qwen3-30B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
8 |
437 |
147.7 |
2,960.0 |
- |
- |
- |
268.8 |
- |
13,715 |
Qwen3-30B-A3B |
LLM |
Vanilla |
NVFP4 |
1 |
61 |
62.9 |
973.6 |
- |
- |
- |
82.1 |
- |
983 |
Qwen3-30B-A3B |
LLM |
Vanilla |
NVFP4 |
8 |
437 |
148.5 |
2,943.4 |
- |
- |
- |
226.5 |
- |
987 |
Qwen3-4B-Instruct-2507 |
LLM |
DFlash |
NVFP4 / NVFP4 |
1 |
57 |
18.8 |
3,043.6 |
- |
- |
- |
98.6 |
2.28 |
1,140 |
Qwen3-4B-Instruct-2507 |
LLM |
DFlash |
NVFP4 / NVFP4 |
8 |
409 |
52.5 |
7,779.0 |
- |
- |
- |
421.9 |
2.19 |
1,150 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
INT4 AWQ |
1 |
57 |
20.1 |
2,850.3 |
- |
- |
- |
80.2 |
- |
1,144 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
INT4 AWQ |
8 |
409 |
110.9 |
3,684.3 |
- |
- |
- |
528.8 |
- |
1,154 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
NVFP4 |
1 |
57 |
19.1 |
3,000.5 |
- |
- |
- |
70.5 |
- |
1,140 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
NVFP4 |
8 |
409 |
51.6 |
7,916.5 |
- |
- |
- |
455.1 |
- |
1,130 |
Qwen3-8B |
LLM |
DFlash |
NVFP4 / NVFP4 |
1 |
61 |
31.7 |
1,928.3 |
- |
- |
- |
77.2 |
3.19 |
1,598 |
Qwen3-8B |
LLM |
DFlash |
NVFP4 / NVFP4 |
8 |
437 |
81.9 |
5,335.8 |
- |
- |
- |
347.5 |
3.05 |
1,592 |
Qwen3-8B |
LLM |
EAGLE3 |
NVFP4 / NVFP4 |
1 |
61 |
27.6 |
2,220.3 |
- |
- |
- |
127.5 |
3.99 |
1,599 |
Qwen3-8B |
LLM |
EAGLE3 |
NVFP4 / NVFP4 |
8 |
437 |
77.4 |
5,647.2 |
- |
- |
- |
417.9 |
4.11 |
1,600 |
Qwen3-8B |
LLM |
Vanilla |
INT4 AWQ |
1 |
61 |
31.7 |
1,931.3 |
- |
- |
- |
45.6 |
- |
1,595 |
Qwen3-8B |
LLM |
Vanilla |
INT4 AWQ |
8 |
437 |
185.9 |
2,351.8 |
- |
- |
- |
309.3 |
- |
1,589 |
Qwen3-8B |
LLM |
Vanilla |
NVFP4 |
1 |
61 |
29.8 |
2,056.4 |
- |
- |
- |
41.2 |
- |
1,598 |
Qwen3-8B |
LLM |
Vanilla |
NVFP4 |
8 |
437 |
81.3 |
5,380.0 |
- |
- |
- |
265.2 |
- |
1,594 |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
292 |
18.5 |
15,836.3 |
14.3 |
265 |
18,612.3 |
150.0 |
- |
1,052 |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,089 |
125.2 |
16,688.8 |
91.8 |
1,896 |
20,650.7 |
847.1 |
- |
1,043 |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
292 |
13.2 |
22,167.6 |
14.2 |
266 |
18,659.2 |
137.1 |
- |
1,054 |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,089 |
65.6 |
31,839.5 |
94.6 |
1,896 |
20,052.6 |
654.2 |
- |
1,053 |
Qwen3-VL-4B-Instruct |
VLM |
EAGLE3 |
NVFP4 / NVFP4 / FP16 |
1 |
292 |
26.0 |
11,256.5 |
14.0 |
266 |
18,994.3 |
227.2 |
4.54 |
1,201 |
Qwen3-VL-4B-Instruct |
VLM |
EAGLE3 |
NVFP4 / NVFP4 / FP16 |
8 |
2,089 |
134.8 |
15,502.5 |
90.0 |
1,896 |
21,057.2 |
628.2 |
4.45 |
1,202 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
292 |
39.4 |
7,422.9 |
14.1 |
265 |
18,882.4 |
80.6 |
- |
1,186 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,089 |
329.8 |
6,334.9 |
90.1 |
1,896 |
21,050.5 |
493.3 |
- |
1,202 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
292 |
26.7 |
10,967.5 |
14.6 |
266 |
18,235.9 |
67.4 |
- |
1,204 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,089 |
143.1 |
14,600.6 |
91.9 |
1,896 |
20,639.7 |
340.1 |
- |
1,201 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
292 |
66.0 |
4,429.9 |
21.5 |
266 |
12,367.1 |
44.0 |
- |
1,649 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,089 |
506.4 |
4,124.9 |
138.8 |
1,896 |
13,659.0 |
303.4 |
- |
1,633 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
292 |
40.8 |
7,174.8 |
21.3 |
265 |
12,469.9 |
42.2 |
- |
1,626 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,089 |
234.6 |
8,905.5 |
138.3 |
1,896 |
13,712.2 |
238.3 |
- |
1,632 |
Qwen3.5-0.8B |
LLM |
Vanilla |
INT4 AWQ |
1 |
62 |
8.7 |
7,099.0 |
- |
- |
- |
235.1 |
- |
899 |
Qwen3.5-0.8B |
LLM |
Vanilla |
INT4 AWQ |
8 |
441 |
25.2 |
17,534.7 |
- |
- |
- |
1,150.8 |
- |
897 |
Qwen3.5-0.8B |
LLM |
Vanilla |
NVFP4 |
1 |
62 |
8.2 |
7,519.7 |
- |
- |
- |
224.6 |
- |
885 |
Qwen3.5-0.8B |
LLM |
Vanilla |
NVFP4 |
8 |
441 |
21.8 |
20,207.1 |
- |
- |
- |
1,130.8 |
- |
890 |
Qwen3.5-0.8B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
296 |
9.7 |
30,463.3 |
4.9 |
265 |
54,464.5 |
299.4 |
2.06 |
929 |
Qwen3.5-0.8B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
2,118 |
51.9 |
40,806.1 |
29.9 |
1,896 |
63,461.6 |
1,006.8 |
2.08 |
1,028 |
Qwen3.5-0.8B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
296 |
12.6 |
23,528.1 |
4.8 |
266 |
54,754.3 |
233.4 |
- |
945 |
Qwen3.5-0.8B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,118 |
66.6 |
31,806.4 |
29.9 |
1,896 |
63,331.4 |
884.1 |
- |
947 |
Qwen3.5-0.8B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
296 |
11.1 |
26,762.0 |
4.8 |
266 |
55,010.8 |
224.1 |
- |
947 |
Qwen3.5-0.8B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,118 |
54.1 |
39,161.6 |
30.0 |
1,896 |
63,138.6 |
912.0 |
- |
947 |
Qwen3.5-27B |
LLM |
Vanilla |
INT4 AWQ |
1 |
62 |
106.6 |
579.6 |
- |
- |
- |
14.2 |
- |
2,839 |
Qwen3.5-27B |
LLM |
Vanilla |
INT4 AWQ |
8 |
441 |
628.2 |
702.3 |
- |
- |
- |
88.6 |
- |
2,836 |
Qwen3.5-27B |
LLM |
Vanilla |
NVFP4 |
1 |
62 |
99.7 |
619.2 |
- |
- |
- |
12.6 |
- |
2,836 |
Qwen3.5-27B |
LLM |
Vanilla |
NVFP4 |
8 |
441 |
304.3 |
1,449.6 |
- |
- |
- |
78.1 |
- |
2,836 |
Qwen3.5-27B |
VLM |
DFlash |
NVFP4 / NVFP4 / FP16 |
1 |
296 |
133.2 |
2,224.9 |
20.2 |
265 |
13,133.9 |
21.0 |
2.56 |
2,880 |
Qwen3.5-27B |
VLM |
DFlash |
NVFP4 / NVFP4 / FP16 |
8 |
2,118 |
886.7 |
2,388.1 |
137.9 |
1,896 |
13,750.6 |
65.2 |
2.52 |
2,877 |
Qwen3.5-27B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
296 |
126.0 |
2,353.6 |
20.3 |
265 |
13,091.4 |
32.2 |
2.82 |
2,880 |
Qwen3.5-27B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
2,118 |
833.4 |
2,541.0 |
131.0 |
1,896 |
14,473.9 |
124.3 |
2.81 |
2,885 |
Qwen3.5-27B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
296 |
224.9 |
1,318.0 |
20.3 |
265 |
13,064.7 |
14.6 |
- |
2,886 |
Qwen3.5-27B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,118 |
1,657.2 |
1,277.8 |
131.3 |
1,896 |
14,445.2 |
82.7 |
- |
2,875 |
Qwen3.5-27B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
296 |
131.6 |
2,252.0 |
20.3 |
266 |
13,079.8 |
13.6 |
- |
2,860 |
Qwen3.5-27B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,118 |
842.1 |
2,514.7 |
131.7 |
1,896 |
14,398.7 |
68.4 |
- |
2,875 |
Qwen3.5-2B |
LLM |
Vanilla |
INT4 AWQ |
1 |
62 |
13.0 |
4,751.4 |
- |
- |
- |
119.5 |
- |
1,384 |
Qwen3.5-2B |
LLM |
Vanilla |
INT4 AWQ |
8 |
441 |
50.0 |
8,817.0 |
- |
- |
- |
611.5 |
- |
1,367 |
Qwen3.5-2B |
LLM |
Vanilla |
NVFP4 |
1 |
62 |
12.6 |
4,901.5 |
- |
- |
- |
117.6 |
- |
1,383 |
Qwen3.5-2B |
LLM |
Vanilla |
NVFP4 |
8 |
441 |
34.5 |
12,780.3 |
- |
- |
- |
683.7 |
- |
1,383 |
Qwen3.5-2B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
296 |
14.9 |
19,911.1 |
13.4 |
265 |
19,762.1 |
205.1 |
2.45 |
1,416 |
Qwen3.5-2B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
2,118 |
78.0 |
27,159.0 |
90.1 |
1,896 |
21,054.9 |
613.7 |
2.38 |
1,420 |
Qwen3.5-2B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
296 |
21.4 |
13,829.8 |
13.3 |
265 |
19,967.5 |
119.0 |
- |
1,432 |
Qwen3.5-2B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,118 |
132.8 |
15,940.4 |
90.7 |
1,896 |
20,903.7 |
538.8 |
- |
1,427 |
Qwen3.5-2B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
296 |
16.3 |
18,177.3 |
13.7 |
265 |
19,342.8 |
115.8 |
- |
1,430 |
Qwen3.5-2B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,118 |
79.6 |
26,603.6 |
88.8 |
1,896 |
21,347.5 |
411.1 |
- |
1,420 |
Qwen3.5-35B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
1 |
62 |
61.0 |
1,012.4 |
- |
- |
- |
47.9 |
- |
15,740 |
Qwen3.5-35B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
8 |
441 |
160.1 |
2,755.9 |
- |
- |
- |
191.2 |
- |
15,773 |
Qwen3.5-35B-A3B |
VLM |
Vanilla |
INT4 GPTQ / FP16 |
1 |
296 |
103.7 |
2,859.1 |
17.9 |
266 |
14,866.2 |
43.0 |
- |
15,810 |
Qwen3.5-35B-A3B |
VLM |
Vanilla |
INT4 GPTQ / FP16 |
8 |
2,118 |
329.4 |
6,429.1 |
128.7 |
1,896 |
14,729.6 |
182.0 |
- |
15,067 |
Qwen3.5-4B |
LLM |
Vanilla |
INT4 AWQ |
1 |
62 |
23.5 |
2,626.5 |
- |
- |
- |
67.2 |
- |
1,623 |
Qwen3.5-4B |
LLM |
Vanilla |
INT4 AWQ |
8 |
441 |
127.5 |
3,459.4 |
- |
- |
- |
352.0 |
- |
1,626 |
Qwen3.5-4B |
LLM |
Vanilla |
NVFP4 |
1 |
62 |
23.0 |
2,680.9 |
- |
- |
- |
63.4 |
- |
1,626 |
Qwen3.5-4B |
LLM |
Vanilla |
NVFP4 |
8 |
441 |
72.3 |
6,104.1 |
- |
- |
- |
356.1 |
- |
1,626 |
Qwen3.5-4B |
VLM |
DFlash |
NVFP4 / NVFP4 / FP16 |
1 |
296 |
34.3 |
8,637.5 |
13.3 |
265 |
20,000.9 |
73.7 |
2.25 |
1,675 |
Qwen3.5-4B |
VLM |
DFlash |
NVFP4 / NVFP4 / FP16 |
8 |
2,118 |
172.0 |
12,309.1 |
85.8 |
1,896 |
22,112.5 |
196.3 |
2.26 |
1,672 |
Qwen3.5-4B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
296 |
30.4 |
9,763.1 |
13.4 |
265 |
19,854.0 |
121.2 |
2.55 |
1,674 |
Qwen3.5-4B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
2,118 |
169.5 |
12,491.8 |
87.2 |
1,896 |
21,740.1 |
385.5 |
2.49 |
1,659 |
Qwen3.5-4B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
296 |
43.7 |
6,784.0 |
13.3 |
266 |
20,026.3 |
68.0 |
- |
1,673 |
Qwen3.5-4B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,118 |
374.2 |
5,658.9 |
86.2 |
1,896 |
22,010.9 |
313.9 |
- |
1,675 |
Qwen3.5-4B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
296 |
33.2 |
8,929.7 |
13.2 |
265 |
20,076.2 |
65.1 |
- |
1,652 |
Qwen3.5-4B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,118 |
171.8 |
12,327.6 |
86.1 |
1,896 |
22,023.7 |
263.6 |
- |
1,670 |
Qwen3.5-9B |
LLM |
Vanilla |
INT4 AWQ |
1 |
62 |
36.6 |
1,687.9 |
- |
- |
- |
38.3 |
- |
2,340 |
Qwen3.5-9B |
LLM |
Vanilla |
INT4 AWQ |
8 |
441 |
199.8 |
2,207.4 |
- |
- |
- |
235.9 |
- |
2,354 |
Qwen3.5-9B |
LLM |
Vanilla |
NVFP4 |
1 |
62 |
36.6 |
1,689.0 |
- |
- |
- |
36.5 |
- |
2,351 |
Qwen3.5-9B |
LLM |
Vanilla |
NVFP4 |
8 |
441 |
99.9 |
4,416.2 |
- |
- |
- |
221.7 |
- |
2,353 |
Qwen3.5-9B |
VLM |
DFlash |
NVFP4 / NVFP4 / FP16 |
1 |
296 |
48.9 |
6,063.5 |
20.5 |
265 |
12,972.6 |
43.5 |
2.36 |
2,402 |
Qwen3.5-9B |
VLM |
DFlash |
NVFP4 / NVFP4 / FP16 |
8 |
2,118 |
270.7 |
7,821.7 |
131.1 |
1,896 |
14,462.1 |
148.5 |
2.39 |
2,400 |
Qwen3.5-9B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
296 |
44.3 |
6,692.9 |
20.1 |
265 |
13,204.3 |
79.3 |
2.69 |
2,403 |
Qwen3.5-9B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
2,118 |
258.9 |
8,179.9 |
130.5 |
1,896 |
14,525.9 |
334.5 |
2.74 |
2,384 |
Qwen3.5-9B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
296 |
72.5 |
4,086.9 |
17.7 |
265 |
15,010.6 |
35.1 |
- |
2,400 |
Qwen3.5-9B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,118 |
533.0 |
3,972.6 |
130.4 |
1,896 |
14,539.7 |
213.5 |
- |
2,401 |
Qwen3.5-9B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
296 |
49.0 |
6,054.0 |
19.9 |
265 |
13,304.5 |
35.0 |
- |
2,402 |
Qwen3.5-9B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,118 |
266.5 |
7,947.1 |
132.0 |
1,896 |
14,369.2 |
180.0 |
- |
2,390 |
Qwen3.6-27B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
296 |
126.6 |
2,342.4 |
20.6 |
265 |
12,898.2 |
31.4 |
2.74 |
2,885 |
Qwen3.6-27B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
2,118 |
823.3 |
2,572.1 |
131.3 |
1,896 |
14,443.7 |
126.3 |
2.76 |
2,869 |
Qwen3.6-35B-A3B |
LLM |
MTP |
NVFP4 / NVFP4 |
1 |
62 |
67.6 |
913.9 |
- |
- |
- |
102.3 |
2.94 |
1,361 |
Qwen3.6-35B-A3B |
LLM |
MTP |
NVFP4 / NVFP4 |
8 |
441 |
166.3 |
2,652.0 |
- |
- |
- |
239.0 |
2.89 |
1,359 |
Qwen3.6-35B-A3B |
LLM |
Vanilla |
NVFP4 |
1 |
62 |
66.9 |
923.8 |
- |
- |
- |
70.2 |
- |
1,359 |
Qwen3.6-35B-A3B |
LLM |
Vanilla |
NVFP4 |
8 |
441 |
164.4 |
2,683.9 |
- |
- |
- |
216.4 |
- |
1,356 |
Qwen3.6-35B-A3B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
296 |
120.1 |
2,469.1 |
20.7 |
265 |
12,810.5 |
96.7 |
2.74 |
1,408 |
Qwen3.6-35B-A3B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
2,118 |
330.2 |
6,413.3 |
129.4 |
1,896 |
14,647.5 |
249.2 |
2.72 |
1,409 |
Qwen3.6-35B-A3B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
296 |
119.0 |
2,491.3 |
17.5 |
265 |
15,176.9 |
69.1 |
- |
1,412 |
Qwen3.6-35B-A3B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,118 |
332.3 |
6,372.0 |
128.2 |
1,896 |
14,786.4 |
226.2 |
- |
1,408 |
v0.10.0 Collection Method#
Engines were built from exported v0.10.0 ONNX artifacts using the build limits in Export and Build Specs.
Runtime throughput was collected with
llm_inference --warmup 10 --dump-profile --profile-output-file <profile.json>using benchmark JSON inputs for each model family.Synthetic component timing was collected with
llm_bench --warmup 3 --iterations 10; prefill uses--input-len 2048and decode uses--past-kv-len 2048.Jetson AGX Thor and DGX Spark runs include the supported NVFP4 and INT4 entries. Jetson AGX Orin, Orin NX, and Orin Nano run the externalized INT4 entries supported by each memory target.
v0.9.0 Results#
SDK Version: TensorRT Edge-LLM 0.9.0 | JetPack: 7.2 | DGX Spark: CUDA 13.0, TensorRT 10.16.1.11 | Devices: Jetson AGX Thor, Jetson AGX Orin 64GB, Jetson Orin NX 16GB, Jetson Orin Nano 8GB, DGX Spark (GB10)
Decode throughput: Runtime
Decode (tok/s)reports generated tokens per second for vanilla decoding and overall accepted-token throughput for speculative decoding.llm_benchBS=8 decode throughput is reported as aggregate batch throughput.
llm_bench Component Performance#
These rows report synthetic llm_bench prefill and decode measurements at the batch sizes shown.
Platform |
Model |
Kind |
Mode |
Precision |
Batch |
Prefill Seq Len |
Prefill E2E (ms) |
Prefill (tok/s) |
Decode Past KV Len |
Decode (tok/s) |
|---|---|---|---|---|---|---|---|---|---|---|
Jetson AGX Thor |
NVIDIA-Nemotron-3-Nano-30B-A3B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
167.3 |
12,238.7 |
2,048 |
76.0 |
Jetson AGX Thor |
NVIDIA-Nemotron-3-Nano-30B-A3B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
1,037 |
1,974.9 |
2,048 |
272.0 |
Jetson AGX Thor |
NVIDIA-Nemotron-3-Nano-4B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
1,150.8 |
1,779.6 |
2,048 |
67.3 |
Jetson AGX Thor |
NVIDIA-Nemotron-3-Nano-4B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
5,352.8 |
382.6 |
2,048 |
433.6 |
Jetson AGX Thor |
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
163.6 |
12,516.2 |
2,048 |
71.2 |
Jetson AGX Thor |
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
992.9 |
2,062.6 |
2,048 |
266.4 |
Jetson AGX Thor |
Qwen3-0.6B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
20.3 |
101,070.8 |
2,048 |
245.4 |
Jetson AGX Thor |
Qwen3-0.6B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
234.6 |
8,730.5 |
2,048 |
715.2 |
Jetson AGX Thor |
Qwen3-1.7B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
29.9 |
68,527.5 |
2,048 |
135.5 |
Jetson AGX Thor |
Qwen3-1.7B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
341.7 |
5,993.2 |
2,048 |
519.2 |
Jetson AGX Thor |
Qwen3-30B-A3B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
137.0 |
14,952.3 |
2,048 |
84.8 |
Jetson AGX Thor |
Qwen3-30B-A3B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
970.8 |
2,109.6 |
2,048 |
249.6 |
Jetson AGX Thor |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
64.7 |
31,645.4 |
2,048 |
73.7 |
Jetson AGX Thor |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
810.0 |
2,528.5 |
2,048 |
349.6 |
Jetson AGX Thor |
Qwen3-8B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
108.4 |
18,896.8 |
2,048 |
44.4 |
Jetson AGX Thor |
Qwen3-8B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
1,214.5 |
1,686.2 |
2,048 |
252.0 |
Jetson AGX Thor |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
30.5 |
67,255.9 |
2,048 |
135.7 |
Jetson AGX Thor |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
343.1 |
5,969.3 |
2,048 |
561.6 |
Jetson AGX Thor |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
64.3 |
31,859.4 |
2,048 |
73.4 |
Jetson AGX Thor |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
813.9 |
2,516.2 |
2,048 |
350.4 |
Jetson AGX Thor |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
108.4 |
18,896.6 |
2,048 |
44.8 |
Jetson AGX Thor |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
1,219.6 |
1,679.2 |
2,048 |
250.4 |
Jetson AGX Thor |
Qwen3.5-0.8B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
36.0 |
56,959.2 |
2,048 |
229.1 |
Jetson AGX Thor |
Qwen3.5-0.8B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
348.3 |
5,880.3 |
2,048 |
1,205.6 |
Jetson AGX Thor |
Qwen3.5-0.8B-LLM |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
36.2 |
56,525.9 |
2,048 |
228.3 |
Jetson AGX Thor |
Qwen3.5-0.8B-LLM |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
347.9 |
5,887.2 |
2,048 |
1,206.4 |
Jetson AGX Thor |
Qwen3.5-27B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
463.3 |
4,420.5 |
2,048 |
14.6 |
Jetson AGX Thor |
Qwen3.5-27B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
4,249.2 |
482.0 |
2,048 |
94.4 |
Jetson AGX Thor |
Qwen3.5-27B-LLM |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
461.9 |
4,433.4 |
2,048 |
14.5 |
Jetson AGX Thor |
Qwen3.5-27B-LLM |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
4,268.3 |
479.8 |
2,048 |
95.2 |
Jetson AGX Thor |
Qwen3.5-2B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
45.3 |
45,230 |
2,048 |
122.4 |
Jetson AGX Thor |
Qwen3.5-2B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
434.8 |
4,710.4 |
2,048 |
756.0 |
Jetson AGX Thor |
Qwen3.5-2B-LLM |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
45.7 |
44,801 |
2,048 |
122.6 |
Jetson AGX Thor |
Qwen3.5-2B-LLM |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
436.3 |
4,694.2 |
2,048 |
757.6 |
Jetson AGX Thor |
Qwen3.5-4B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
101.1 |
20,248.4 |
2,048 |
68.0 |
Jetson AGX Thor |
Qwen3.5-4B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
996.5 |
2,055.2 |
2,048 |
395.2 |
Jetson AGX Thor |
Qwen3.5-4B-LLM |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
101.9 |
20,106.1 |
2,048 |
67.7 |
Jetson AGX Thor |
Qwen3.5-4B-LLM |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
992.0 |
2,064.5 |
2,048 |
394.4 |
Jetson AGX Thor |
Qwen3.5-9B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
139.0 |
14,738.3 |
2,048 |
40.1 |
Jetson AGX Thor |
Qwen3.5-9B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
1,406 |
1,456.6 |
2,048 |
261.6 |
Jetson AGX Thor |
Qwen3.5-9B-LLM |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
138.4 |
14,800 |
2,048 |
39.7 |
Jetson AGX Thor |
Qwen3.5-9B-LLM |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
1,414.5 |
1,447.8 |
2,048 |
260.0 |
Jetson AGX Thor |
Qwen3.6-35B-A3B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
195.0 |
10,501.2 |
2,048 |
84.1 |
Jetson AGX Thor |
Qwen3.6-35B-A3B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
1,212.8 |
1,688.7 |
2,048 |
257.6 |
DGX Spark (GB10) |
NVIDIA-Nemotron-3-Nano-30B-A3B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
312.3 |
6,557.2 |
2,048 |
69.4 |
DGX Spark (GB10) |
NVIDIA-Nemotron-3-Nano-30B-A3B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
2,162.2 |
947.2 |
2,048 |
248.0 |
DGX Spark (GB10) |
NVIDIA-Nemotron-3-Nano-4B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
671.8 |
3,048.4 |
2,048 |
56.8 |
DGX Spark (GB10) |
NVIDIA-Nemotron-3-Nano-4B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
3,103.6 |
659.9 |
2,048 |
392.8 |
DGX Spark (GB10) |
Qwen3-0.6B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
34.5 |
59,320.0 |
2,048 |
202.6 |
DGX Spark (GB10) |
Qwen3-0.6B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
310.5 |
6,595.2 |
2,048 |
620.8 |
DGX Spark (GB10) |
Qwen3-1.7B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
60.5 |
33,860.2 |
2,048 |
113.8 |
DGX Spark (GB10) |
Qwen3-1.7B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
505.9 |
4,047.9 |
2,048 |
479.2 |
DGX Spark (GB10) |
Qwen3-30B-A3B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
292.8 |
6,994.7 |
2,048 |
73.3 |
DGX Spark (GB10) |
Qwen3-30B-A3B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
2,133.4 |
960.0 |
2,048 |
214.4 |
DGX Spark (GB10) |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
128.4 |
15,948.1 |
2,048 |
62.6 |
DGX Spark (GB10) |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
1,084.9 |
1,887.8 |
2,048 |
298.4 |
DGX Spark (GB10) |
Qwen3-8B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
192.4 |
10,646.5 |
2,048 |
37.4 |
DGX Spark (GB10) |
Qwen3-8B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
1,509.9 |
1,356.4 |
2,048 |
216.8 |
DGX Spark (GB10) |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
58.9 |
34,797.5 |
2,048 |
116.0 |
DGX Spark (GB10) |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
530.0 |
3,864.2 |
2,048 |
476.0 |
DGX Spark (GB10) |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
136.6 |
14,993.1 |
2,048 |
61.3 |
DGX Spark (GB10) |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
1,094.1 |
1,871.8 |
2,048 |
299.2 |
DGX Spark (GB10) |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
198.9 |
10,297.3 |
2,048 |
37.4 |
DGX Spark (GB10) |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
1,519.6 |
1,347.7 |
2,048 |
220.0 |
DGX Spark (GB10) |
Qwen3.5-0.8B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
302.7 |
6,764.7 |
2,048 |
189.5 |
DGX Spark (GB10) |
Qwen3.5-0.8B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
759.6 |
2,696.2 |
2,048 |
1,073.6 |
DGX Spark (GB10) |
Qwen3.5-0.8B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
299.3 |
6,842.6 |
2,048 |
197.7 |
DGX Spark (GB10) |
Qwen3.5-0.8B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
763.3 |
2,683.0 |
2,048 |
1,050.4 |
DGX Spark (GB10) |
Qwen3.5-27B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
1,291.1 |
1,586.2 |
2,048 |
12.5 |
DGX Spark (GB10) |
Qwen3.5-27B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
7,144.6 |
286.6 |
2,048 |
84.8 |
DGX Spark (GB10) |
Qwen3.5-27B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
1,306.0 |
1,568.2 |
2,048 |
12.4 |
DGX Spark (GB10) |
Qwen3.5-27B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
7,211.7 |
284.0 |
2,048 |
83.2 |
DGX Spark (GB10) |
Qwen3.5-2B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
327.0 |
6,263.6 |
2,048 |
106.9 |
DGX Spark (GB10) |
Qwen3.5-2B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
923.6 |
2,217.4 |
2,048 |
666.4 |
DGX Spark (GB10) |
Qwen3.5-2B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
325.9 |
6,283.7 |
2,048 |
105.5 |
DGX Spark (GB10) |
Qwen3.5-2B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
907.1 |
2,257.8 |
2,048 |
666.4 |
DGX Spark (GB10) |
Qwen3.5-4B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
498.5 |
4,108.7 |
2,048 |
56.7 |
DGX Spark (GB10) |
Qwen3.5-4B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
2,099.3 |
975.6 |
2,048 |
340.8 |
DGX Spark (GB10) |
Qwen3.5-4B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
498.4 |
4,109.3 |
2,048 |
58.2 |
DGX Spark (GB10) |
Qwen3.5-4B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
2,130.9 |
961.1 |
2,048 |
344.0 |
DGX Spark (GB10) |
Qwen3.5-9B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
556.6 |
3,679.8 |
2,048 |
34.7 |
DGX Spark (GB10) |
Qwen3.5-9B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
2,511.7 |
815.4 |
2,048 |
228.0 |
DGX Spark (GB10) |
Qwen3.5-9B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
555.0 |
3,690.1 |
2,048 |
34.1 |
DGX Spark (GB10) |
Qwen3.5-9B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
2,513.4 |
814.8 |
2,048 |
231.2 |
DGX Spark (GB10) |
Qwen3.6-35B-A3B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
728.8 |
2,809.9 |
2,048 |
67.0 |
DGX Spark (GB10) |
Qwen3.6-35B-A3B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
3,069.1 |
667.3 |
2,048 |
214.4 |
DGX Spark (GB10) |
Qwen3.6-35B-A3B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
728.0 |
2,813.1 |
2,048 |
67.1 |
DGX Spark (GB10) |
Qwen3.6-35B-A3B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
3,075.0 |
666.0 |
2,048 |
212.0 |
DGX Spark (GB10) |
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
311.7 |
6,570.7 |
2,048 |
63.8 |
DGX Spark (GB10) |
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
2,141.4 |
956.4 |
2,048 |
245.6 |
Runtime Performance Dashboard#
Runtime rows are split by device and include batch size, prefill sequence length/time, visual encoder timing for VLMs, decode throughput, speculative acceptance rate, and peak GPU memory.
Jetson AGX Thor#
Model |
Kind |
Mode |
Precision |
Dataset |
Batch |
Prefill Seq Len |
Prefill Time (ms) |
Prefill (tok/s) |
ViT Time (ms) |
ViT Tok/Run |
ViT (tok/s) |
Decode (tok/s) |
Accept Rate |
GPU Mem (MB) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
NVIDIA-Nemotron-3-Nano-30B-A3B |
LLM |
Vanilla |
NVFP4 |
- |
1 |
66 |
73.9 |
891.0 |
- |
- |
- |
77.3 |
- |
18,754 |
NVIDIA-Nemotron-3-Nano-30B-A3B |
LLM |
Vanilla |
NVFP4 |
- |
8 |
470 |
157.1 |
2,992.8 |
- |
- |
- |
225.9 |
- |
18,721 |
NVIDIA-Nemotron-3-Nano-4B |
LLM |
Vanilla |
NVFP4 |
- |
1 |
66 |
35.4 |
1,858.9 |
- |
- |
- |
67.9 |
- |
3,546 |
NVIDIA-Nemotron-3-Nano-4B |
LLM |
Vanilla |
NVFP4 |
- |
8 |
470 |
147.3 |
3,192.7 |
- |
- |
- |
359.5 |
- |
3,556 |
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
1 |
1,699 |
159.4 |
10,658 |
127.3 |
1,664 |
13,070.9 |
72.6 |
- |
18,928 |
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
8 |
12,136 |
806.8 |
15,042.2 |
932.8 |
11,886 |
12,742.1 |
183.5 |
- |
18,979 |
Qwen2.5-VL-7B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
1 |
376 |
177.4 |
2,121.1 |
32.8 |
349 |
10,661.3 |
84.6 |
4.91 |
3,320 |
Qwen2.5-VL-7B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
8 |
2,688 |
1,333.9 |
2,014.8 |
260.5 |
2,495 |
9,575.1 |
126.7 |
4.84 |
3,332 |
Qwen2.5-VL-7B-Instruct |
VLM |
EAGLE3 |
NVFP4 / NVFP4 / FP16 |
COCO |
1 |
376 |
23.2 |
16,193.8 |
32.7 |
349 |
10,685 |
188.4 |
4.77 |
4,260 |
Qwen2.5-VL-7B-Instruct |
VLM |
EAGLE3 |
NVFP4 / NVFP4 / FP16 |
COCO |
8 |
2,688 |
180.0 |
14,931.8 |
259.5 |
2,495 |
9,613.4 |
531.1 |
4.73 |
4,288 |
Qwen3-0.6B |
LLM |
Vanilla |
NVFP4 |
- |
1 |
61 |
8.0 |
7,672.9 |
- |
- |
- |
303.1 |
- |
966 |
Qwen3-0.6B |
LLM |
Vanilla |
NVFP4 |
- |
8 |
437 |
19.6 |
22,337.1 |
- |
- |
- |
1,569.5 |
- |
961 |
Qwen3-1.7B |
LLM |
Vanilla |
NVFP4 |
- |
1 |
61 |
12.0 |
5,110 |
- |
- |
- |
154.2 |
- |
1,771 |
Qwen3-1.7B |
LLM |
Vanilla |
NVFP4 |
- |
8 |
437 |
29.6 |
14,790.2 |
- |
- |
- |
828.7 |
- |
1,806 |
Qwen3-1.7B |
LLM |
EAGLE3 |
NVFP4 / NVFP4 |
- |
1 |
61 |
10.0 |
6,143.8 |
- |
- |
- |
351.6 |
3.04 |
1,355 |
Qwen3-1.7B |
LLM |
EAGLE3 |
NVFP4 / NVFP4 |
- |
8 |
437 |
27.7 |
15,768.7 |
- |
- |
- |
950.4 |
3.06 |
1,365 |
Qwen3-30B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
- |
1 |
61 |
64.1 |
955.4 |
- |
- |
- |
85.8 |
- |
14,305 |
Qwen3-30B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
- |
8 |
437 |
263.7 |
1,657.9 |
- |
- |
- |
232.1 |
- |
14,314 |
Qwen3-30B-A3B |
LLM |
Vanilla |
NVFP4 |
- |
1 |
61 |
57.1 |
1,071.8 |
- |
- |
- |
90.4 |
- |
17,112 |
Qwen3-30B-A3B |
LLM |
Vanilla |
NVFP4 |
- |
8 |
437 |
107.1 |
4,081.1 |
- |
- |
- |
252.8 |
- |
17,092 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
INT4 AWQ |
- |
1 |
57 |
33.5 |
1,710.1 |
- |
- |
- |
79.6 |
- |
1,819 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
INT4 AWQ |
- |
8 |
409 |
229.8 |
1,778 |
- |
- |
- |
328.6 |
- |
1,808 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
NVFP4 |
- |
1 |
57 |
20.5 |
2,790 |
- |
- |
- |
80.5 |
- |
3,138 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
NVFP4 |
- |
8 |
409 |
53.9 |
7,586.7 |
- |
- |
- |
494.9 |
- |
3,151 |
Qwen3-4B-Instruct-2507 |
LLM |
DFlash |
NVFP4 / NVFP4 |
- |
1 |
57 |
20.6 |
2,772.8 |
- |
- |
- |
102.3 |
2.26 |
3,141 |
Qwen3-4B-Instruct-2507 |
LLM |
DFlash |
NVFP4 / NVFP4 |
- |
8 |
409 |
53.8 |
7,597 |
- |
- |
- |
462.1 |
2.24 |
3,148 |
Qwen3-8B |
LLM |
Vanilla |
INT4 AWQ |
- |
1 |
61 |
56.0 |
1,093.1 |
- |
- |
- |
49.3 |
- |
3,157 |
Qwen3-8B |
LLM |
Vanilla |
INT4 AWQ |
- |
8 |
437 |
426.5 |
1,025.1 |
- |
- |
- |
191.8 |
- |
3,166 |
Qwen3-8B |
LLM |
Vanilla |
NVFP4 |
- |
1 |
61 |
30.4 |
2,011.8 |
- |
- |
- |
46.8 |
- |
5,361 |
Qwen3-8B |
LLM |
Vanilla |
NVFP4 |
- |
8 |
437 |
74.1 |
5,901.8 |
- |
- |
- |
283.1 |
- |
5,374 |
Qwen3-8B |
LLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ |
- |
1 |
61 |
55.8 |
1,097.7 |
- |
- |
- |
77.1 |
4.22 |
3,146 |
Qwen3-8B |
LLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ |
- |
8 |
437 |
426.2 |
1,025.7 |
- |
- |
- |
107.7 |
4.20 |
3,153 |
Qwen3-8B |
LLM |
EAGLE3 |
NVFP4 / NVFP4 |
- |
1 |
61 |
26.7 |
2,291.6 |
- |
- |
- |
160.5 |
4.24 |
4,513 |
Qwen3-8B |
LLM |
EAGLE3 |
NVFP4 / NVFP4 |
- |
8 |
437 |
71.1 |
6,149.4 |
- |
- |
- |
488.5 |
4.12 |
4,523 |
Qwen3-8B |
LLM |
DFlash |
NVFP4 / NVFP4 |
- |
1 |
61 |
29.5 |
2,073.7 |
- |
- |
- |
87.1 |
3.24 |
5,363 |
Qwen3-8B |
LLM |
DFlash |
NVFP4 / NVFP4 |
- |
8 |
437 |
74.8 |
5,848.2 |
- |
- |
- |
378.5 |
3.13 |
5,379 |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
1 |
292 |
10.7 |
27,311.9 |
11.7 |
266 |
22,678.4 |
151.9 |
- |
1,845 |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
8 |
2,089 |
53.2 |
39,302.3 |
75.6 |
1,896 |
25,068 |
698.8 |
- |
1,853 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
1 |
292 |
85.2 |
3,433.5 |
11.7 |
266 |
22,731.2 |
79.0 |
- |
1,847 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
8 |
2,089 |
613.0 |
3,408 |
76.0 |
1,896 |
24,964.3 |
273.1 |
- |
1,850 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
1 |
292 |
20.4 |
14,327 |
12.2 |
265 |
21,831.3 |
79.4 |
- |
3,182 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
8 |
2,089 |
107.0 |
19,531 |
76.0 |
1,896 |
24,951.1 |
377.9 |
- |
3,193 |
Qwen3-VL-4B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
1 |
292 |
85.2 |
3,431.1 |
11.7 |
266 |
22,640.1 |
143.0 |
4.95 |
1,847 |
Qwen3-VL-4B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
8 |
2,089 |
613.3 |
3,406.2 |
76.2 |
1,896 |
24,895.9 |
212.9 |
4.91 |
1,846 |
Qwen3-VL-4B-Instruct |
VLM |
EAGLE3 |
NVFP4 / NVFP4 / FP16 |
COCO |
1 |
292 |
18.3 |
15,978.8 |
11.7 |
265 |
22,742.1 |
258.3 |
4.48 |
2,660 |
Qwen3-VL-4B-Instruct |
VLM |
EAGLE3 |
NVFP4 / NVFP4 / FP16 |
COCO |
8 |
2,089 |
105.0 |
19,894.8 |
75.5 |
1,896 |
25,132.1 |
676.7 |
4.57 |
2,680 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
1 |
292 |
156.5 |
1,869.1 |
15.8 |
265 |
16,778.3 |
48.9 |
- |
3,204 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
8 |
2,089 |
1,113.2 |
1,876.6 |
108.1 |
1,896 |
17,546.9 |
184.5 |
- |
3,191 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
1 |
292 |
30.2 |
9,681.1 |
15.8 |
265 |
16,854.4 |
47.1 |
- |
5,403 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
8 |
2,089 |
175.5 |
11,904.6 |
108.5 |
1,896 |
17,474.1 |
246.9 |
- |
5,424 |
Qwen3-VL-8B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
1 |
292 |
156.5 |
1,868.5 |
15.7 |
265 |
16,899 |
70.8 |
3.93 |
3,199 |
Qwen3-VL-8B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
8 |
2,089 |
1,114.1 |
1,875.1 |
108.1 |
1,896 |
17,549.2 |
99.5 |
3.93 |
3,193 |
Qwen3.5-0.8B |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
1 |
296 |
9.4 |
31,415 |
3.9 |
265 |
68,378.8 |
234.6 |
- |
1,239 |
Qwen3.5-0.8B |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
8 |
2,118 |
51.6 |
41,042.8 |
26.9 |
1,896 |
70,417.5 |
976.7 |
- |
1,223 |
Qwen3.5-0.8B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
COCO |
1 |
296 |
8.1 |
36,604.8 |
3.9 |
266 |
68,283.8 |
377.4 |
2.09 |
1,016 |
Qwen3.5-0.8B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
COCO |
8 |
2,118 |
49.9 |
42,408 |
26.9 |
1,896 |
70,395.1 |
1,118.1 |
2.09 |
1,135 |
Qwen3.5-0.8B-LLM |
LLM |
Vanilla |
NVFP4 |
- |
1 |
62 |
9.2 |
6,695.1 |
- |
- |
- |
235.1 |
- |
1,184 |
Qwen3.5-0.8B-LLM |
LLM |
Vanilla |
NVFP4 |
- |
8 |
441 |
26.2 |
16,871.5 |
- |
- |
- |
1,230.9 |
- |
1,153 |
Qwen3.5-27B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
1 |
296 |
567.0 |
522.8 |
14.6 |
266 |
18,137.7 |
15.8 |
- |
8,975 |
Qwen3.5-27B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
8 |
2,118 |
4,498 |
470.8 |
102.5 |
1,896 |
18,502 |
56.9 |
- |
8,982 |
Qwen3.5-27B |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
1 |
296 |
104.0 |
2,851.2 |
14.5 |
265 |
18,265.8 |
14.8 |
- |
16,040 |
Qwen3.5-27B |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
8 |
2,118 |
654.1 |
3,237.4 |
102.6 |
1,896 |
18,486.3 |
76.0 |
- |
16,081 |
Qwen3.5-27B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
1 |
296 |
567.0 |
522.9 |
14.6 |
265 |
18,134 |
27.9 |
2.85 |
8,972 |
Qwen3.5-27B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
8 |
2,118 |
4,498.2 |
470.8 |
102.3 |
1,896 |
18,538.5 |
92.3 |
2.84 |
9,006 |
Qwen3.5-27B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
COCO |
1 |
296 |
94.8 |
3,126.6 |
14.7 |
265 |
18,122.6 |
37.6 |
2.84 |
14,309 |
Qwen3.5-27B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
COCO |
8 |
2,118 |
645.5 |
3,280.4 |
102.8 |
1,896 |
18,448.6 |
152.9 |
2.85 |
14,353 |
Qwen3.5-27B |
VLM |
DFlash |
NVFP4 / NVFP4 / FP16 |
COCO |
1 |
296 |
103.6 |
2,862.1 |
14.6 |
265 |
18,239.9 |
21.7 |
2.59 |
16,063 |
Qwen3.5-27B |
VLM |
DFlash |
NVFP4 / NVFP4 / FP16 |
COCO |
8 |
2,118 |
656.0 |
3,227.8 |
103.0 |
1,896 |
18,416.8 |
55.6 |
2.51 |
16,077 |
Qwen3.5-27B-LLM |
LLM |
Vanilla |
INT4 AWQ |
- |
1 |
62 |
172.1 |
358.9 |
- |
- |
- |
15.8 |
- |
8,932 |
Qwen3.5-27B-LLM |
LLM |
Vanilla |
INT4 AWQ |
- |
8 |
441 |
1,717 |
256.9 |
- |
- |
- |
60.2 |
- |
8,938 |
Qwen3.5-27B-LLM |
LLM |
Vanilla |
NVFP4 |
- |
1 |
62 |
90.5 |
682.2 |
- |
- |
- |
14.7 |
- |
15,983 |
Qwen3.5-27B-LLM |
LLM |
Vanilla |
NVFP4 |
- |
8 |
441 |
276.6 |
1,594.7 |
- |
- |
- |
88.5 |
- |
16,018 |
Qwen3.5-2B |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
1 |
296 |
13.8 |
21,423.1 |
11.2 |
265 |
23,721.7 |
125.3 |
- |
2,186 |
Qwen3.5-2B |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
8 |
2,118 |
69.4 |
30,536.6 |
72.8 |
1,896 |
26,044.4 |
475.1 |
- |
2,188 |
Qwen3.5-2B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
1 |
296 |
42.0 |
7,067.5 |
11.1 |
265 |
23,820.9 |
131.0 |
2.43 |
1,579 |
Qwen3.5-2B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
8 |
2,118 |
347.2 |
6,099.6 |
73.3 |
1,896 |
25,874.3 |
483.5 |
2.41 |
1,581 |
Qwen3.5-2B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
COCO |
1 |
296 |
11.4 |
26,036.1 |
11.2 |
265 |
23,798.3 |
254.5 |
2.40 |
1,475 |
Qwen3.5-2B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
COCO |
8 |
2,118 |
65.4 |
32,395.7 |
72.5 |
1,896 |
26,172.8 |
730.4 |
2.33 |
1,502 |
Qwen3.5-2B-LLM |
LLM |
Vanilla |
NVFP4 |
- |
1 |
62 |
13.9 |
4,453.7 |
- |
- |
- |
123.5 |
- |
2,134 |
Qwen3.5-2B-LLM |
LLM |
Vanilla |
NVFP4 |
- |
8 |
441 |
34.5 |
12,782.4 |
- |
- |
- |
729.5 |
- |
2,161 |
Qwen3.5-35B-A3B |
VLM |
Vanilla |
INT4 GPTQ / FP16 |
COCO |
1 |
296 |
109.8 |
2,700.1 |
14.5 |
265 |
18,333.6 |
47.9 |
- |
15,868 |
Qwen3.5-35B-A3B |
VLM |
Vanilla |
INT4 GPTQ / FP16 |
COCO |
8 |
2,118 |
458.2 |
4,621.5 |
101.6 |
1,896 |
18,655.7 |
196.3 |
- |
15,872 |
Qwen3.5-35B-A3B-LLM |
LLM |
Vanilla |
INT4 GPTQ |
- |
1 |
62 |
66.6 |
928.0 |
- |
- |
- |
48.2 |
- |
15,829 |
Qwen3.5-35B-A3B-LLM |
LLM |
Vanilla |
INT4 GPTQ |
- |
8 |
441 |
212.2 |
2,079.4 |
- |
- |
- |
203.1 |
- |
15,823 |
Qwen3.5-4B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
1 |
296 |
98.6 |
3,007 |
11.0 |
265 |
24,210.2 |
67.5 |
- |
1,731 |
Qwen3.5-4B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
8 |
2,118 |
740.8 |
2,858.4 |
72.7 |
1,896 |
26,080.2 |
243.1 |
- |
1,748 |
Qwen3.5-4B |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
1 |
296 |
26.1 |
11,347 |
10.9 |
265 |
24,398 |
69.4 |
- |
3,635 |
Qwen3.5-4B |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
8 |
2,118 |
142.1 |
14,906.9 |
72.9 |
1,896 |
26,020.4 |
271.3 |
- |
3,636 |
Qwen3.5-4B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
1 |
296 |
98.6 |
3,005.8 |
10.9 |
265 |
24,308.6 |
83.1 |
2.55 |
1,838 |
Qwen3.5-4B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
8 |
2,118 |
840.9 |
2,518.2 |
73.1 |
1,896 |
25,935.5 |
278.7 |
2.55 |
1,835 |
Qwen3.5-4B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
COCO |
1 |
296 |
22.6 |
13,107.3 |
11.0 |
266 |
24,251.3 |
144.1 |
2.55 |
2,749 |
Qwen3.5-4B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
COCO |
8 |
2,118 |
139.6 |
15,171.4 |
73.2 |
1,896 |
25,895.5 |
439.8 |
2.50 |
2,774 |
Qwen3.5-4B |
VLM |
DFlash |
NVFP4 / NVFP4 / FP16 |
COCO |
1 |
296 |
26.3 |
11,277.7 |
11.2 |
265 |
23,704.3 |
73.8 |
2.27 |
3,631 |
Qwen3.5-4B |
VLM |
DFlash |
NVFP4 / NVFP4 / FP16 |
COCO |
8 |
2,118 |
143.6 |
14,750.3 |
73.4 |
1,896 |
25,828.5 |
169.0 |
2.29 |
3,627 |
Qwen3.5-4B-LLM |
LLM |
Vanilla |
INT4 AWQ |
- |
1 |
62 |
37.4 |
1,651.5 |
- |
- |
- |
68.3 |
- |
1,691 |
Qwen3.5-4B-LLM |
LLM |
Vanilla |
INT4 AWQ |
- |
8 |
441 |
287.5 |
1,534.3 |
- |
- |
- |
274.0 |
- |
1,674 |
Qwen3.5-4B-LLM |
LLM |
Vanilla |
NVFP4 |
- |
1 |
62 |
24.2 |
2,548.5 |
- |
- |
- |
69.1 |
- |
3,571 |
Qwen3.5-4B-LLM |
LLM |
Vanilla |
NVFP4 |
- |
8 |
441 |
69.1 |
6,382.2 |
- |
- |
- |
386.2 |
- |
3,561 |
Qwen3.5-9B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
1 |
296 |
173.8 |
1,705.9 |
14.6 |
265 |
18,164.8 |
42.5 |
- |
2,896 |
Qwen3.5-9B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
8 |
2,118 |
1,354.8 |
1,563 |
101.6 |
1,896 |
18,664.1 |
161.7 |
- |
2,922 |
Qwen3.5-9B |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
1 |
296 |
37.6 |
7,895.3 |
14.5 |
265 |
18,291.2 |
40.8 |
- |
6,137 |
Qwen3.5-9B |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
8 |
2,118 |
211.4 |
10,016 |
101.1 |
1,896 |
18,757.2 |
187.2 |
- |
6,150 |
Qwen3.5-9B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
1 |
296 |
173.8 |
1,705.4 |
14.6 |
265 |
18,240.9 |
59.2 |
2.77 |
2,901 |
Qwen3.5-9B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
8 |
2,118 |
1,355.5 |
1,562.2 |
100.9 |
1,896 |
18,798.2 |
225.0 |
2.78 |
2,920 |
Qwen3.5-9B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
COCO |
1 |
296 |
31.9 |
9,294 |
14.7 |
265 |
18,097.9 |
96.7 |
2.70 |
4,754 |
Qwen3.5-9B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
COCO |
8 |
2,118 |
204.8 |
10,338.5 |
101.9 |
1,896 |
18,616.8 |
396.5 |
2.73 |
4,768 |
Qwen3.5-9B |
VLM |
DFlash |
NVFP4 / NVFP4 / FP16 |
COCO |
1 |
296 |
37.6 |
7,885.8 |
14.5 |
265 |
18,263.8 |
47.1 |
2.42 |
6,145 |
Qwen3.5-9B |
VLM |
DFlash |
NVFP4 / NVFP4 / FP16 |
COCO |
8 |
2,118 |
212.6 |
9,961.9 |
101.7 |
1,896 |
18,653.1 |
132.4 |
2.34 |
6,161 |
Qwen3.5-9B-LLM |
LLM |
Vanilla |
INT4 AWQ |
- |
1 |
62 |
60.7 |
1,017.2 |
- |
- |
- |
42.3 |
- |
2,833 |
Qwen3.5-9B-LLM |
LLM |
Vanilla |
INT4 AWQ |
- |
8 |
441 |
521.4 |
846.1 |
- |
- |
- |
167.2 |
- |
2,865 |
Qwen3.5-9B-LLM |
LLM |
Vanilla |
NVFP4 |
- |
1 |
62 |
34.2 |
1,803.9 |
- |
- |
- |
40.8 |
- |
6,074 |
Qwen3.5-9B-LLM |
LLM |
Vanilla |
NVFP4 |
- |
8 |
441 |
92.4 |
4,774.8 |
- |
- |
- |
242.0 |
- |
6,104 |
Qwen3.6-27B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
1 |
296 |
567.0 |
522.9 |
14.7 |
265 |
18,107.5 |
27.3 |
2.78 |
8,969 |
Qwen3.6-27B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
8 |
2,118 |
4,498.4 |
470.7 |
102.1 |
1,896 |
18,573.5 |
90.8 |
2.81 |
8,976 |
Qwen3.6-27B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
COCO |
1 |
296 |
96.8 |
3,062.1 |
14.7 |
265 |
18,093.5 |
37.1 |
2.82 |
14,328 |
Qwen3.6-27B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
COCO |
8 |
2,118 |
645.8 |
3,279 |
102.2 |
1,896 |
18,559.2 |
143.3 |
2.81 |
14,335 |
Qwen3.6-35B-A3B |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
1 |
296 |
88.8 |
3,339.5 |
14.5 |
265 |
18,334.9 |
87.6 |
- |
19,472 |
Qwen3.6-35B-A3B |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
8 |
2,118 |
229.3 |
9,236.3 |
101.4 |
1,896 |
18,696.7 |
251.1 |
- |
19,468 |
Qwen3.6-35B-A3B |
VLM |
DFlash |
NVFP4 / NVFP4 / FP16 |
COCO |
1 |
296 |
89.7 |
3,304.7 |
14.5 |
265 |
18,318.7 |
42.2 |
1.62 |
19,389 |
Qwen3.6-35B-A3B |
VLM |
DFlash |
NVFP4 / NVFP4 / FP16 |
COCO |
8 |
2,118 |
228.8 |
9,253.9 |
100.5 |
1,896 |
18,862 |
86.9 |
1.59 |
19,474 |
Jetson AGX Orin (64GB)#
Model |
Kind |
Mode |
Precision |
Dataset |
Batch |
Prefill Seq Len |
Prefill Time (ms) |
Prefill (tok/s) |
ViT Time (ms) |
ViT Tok/Run |
ViT (tok/s) |
Decode (tok/s) |
Accept Rate |
GPU Mem (MB) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Qwen2.5-VL-7B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
1 |
376 |
260.9 |
1,442.1 |
89.2 |
349 |
3,915.4 |
64.4 |
4.81 |
12,141 |
Qwen2.5-VL-7B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
8 |
2,688 |
1,936 |
1,388.2 |
635.0 |
2,495 |
3,928.8 |
85.4 |
4.78 |
14,078 |
Qwen3-0.6B |
LLM |
Vanilla |
INT4 AWQ |
- |
1 |
61 |
14.8 |
4,124.1 |
- |
- |
- |
186.4 |
- |
1,931 |
Qwen3-0.6B |
LLM |
Vanilla |
INT4 AWQ |
- |
8 |
437 |
50.9 |
8,588.1 |
- |
- |
- |
738.2 |
- |
4,043 |
Qwen3-1.7B |
LLM |
Vanilla |
INT4 AWQ |
- |
1 |
61 |
20.4 |
2,995.1 |
- |
- |
- |
98.6 |
- |
3,302 |
Qwen3-1.7B |
LLM |
Vanilla |
INT4 AWQ |
- |
8 |
437 |
134.2 |
3,257.9 |
- |
- |
- |
465.5 |
- |
5,602 |
Qwen3-1.7B |
LLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ |
- |
1 |
61 |
20.5 |
2,992.9 |
- |
- |
- |
128.2 |
3.18 |
3,395 |
Qwen3-1.7B |
LLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ |
- |
8 |
437 |
134.8 |
3,243.1 |
- |
- |
- |
215.3 |
3.18 |
6,714 |
Qwen3-30B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
- |
1 |
61 |
91.0 |
672.4 |
- |
- |
- |
56.7 |
- |
26,077 |
Qwen3-30B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
- |
8 |
437 |
389.2 |
1,123.2 |
- |
- |
- |
163.3 |
- |
25,304 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
INT4 AWQ |
- |
1 |
57 |
42.6 |
1,342.4 |
- |
- |
- |
53.1 |
- |
4,911 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
INT4 AWQ |
- |
8 |
409 |
323.6 |
1,262.8 |
- |
- |
- |
225.8 |
- |
8,119 |
Qwen3-8B |
LLM |
Vanilla |
INT4 AWQ |
- |
1 |
61 |
65.7 |
931.2 |
- |
- |
- |
32.5 |
- |
8,221 |
Qwen3-8B |
LLM |
Vanilla |
INT4 AWQ |
- |
8 |
437 |
592.8 |
737.4 |
- |
- |
- |
151.5 |
- |
11,134 |
Qwen3-8B |
LLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ |
- |
1 |
61 |
65.6 |
933.3 |
- |
- |
- |
59.3 |
4.22 |
8,344 |
Qwen3-8B |
LLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ |
- |
8 |
437 |
593.4 |
736.7 |
- |
- |
- |
72.6 |
4.21 |
12,914 |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
1 |
292 |
50.0 |
5,853 |
39.0 |
265 |
6,815.9 |
98.4 |
- |
8,140 |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
8 |
2,089 |
355.8 |
5,870.7 |
256.8 |
1,896 |
7,383.3 |
372.3 |
- |
10,246 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
1 |
292 |
123.7 |
2,364 |
39.3 |
265 |
6,749.4 |
52.4 |
- |
9,516 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
8 |
2,089 |
886.2 |
2,357.2 |
257.6 |
1,896 |
7,360.4 |
188.7 |
- |
12,436 |
Qwen3-VL-4B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
1 |
292 |
123.9 |
2,360.7 |
39.2 |
265 |
6,765.4 |
96.2 |
4.95 |
10,231 |
Qwen3-VL-4B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
8 |
2,089 |
889.5 |
2,348.6 |
258.2 |
1,896 |
7,342.6 |
141.4 |
4.88 |
13,573 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
1 |
292 |
225.1 |
1,299.1 |
57.2 |
265 |
4,644.4 |
32.3 |
- |
13,030 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
8 |
2,089 |
1,598.1 |
1,307.2 |
373.2 |
1,896 |
5,080.7 |
144.5 |
- |
15,904 |
Qwen3-VL-8B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
1 |
292 |
225.8 |
1,295.3 |
56.9 |
265 |
4,668.9 |
54.8 |
3.94 |
13,585 |
Qwen3-VL-8B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
8 |
2,089 |
1,603.8 |
1,302.6 |
372.5 |
1,896 |
5,090.6 |
67.0 |
3.92 |
17,381 |
Qwen3.5-0.8B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
1 |
296 |
96.8 |
3,062.6 |
13.2 |
266 |
20,176.9 |
146.0 |
- |
4,488 |
Qwen3.5-0.8B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
8 |
2,118 |
440.5 |
4,806.9 |
84.2 |
1,896 |
22,521 |
574.3 |
- |
4,992 |
Qwen3.5-0.8B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
1 |
296 |
97.3 |
3,047.3 |
13.2 |
265 |
20,168.1 |
130.7 |
2.06 |
5,108 |
Qwen3.5-0.8B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
8 |
2,118 |
440.5 |
4,807.6 |
83.9 |
1,896 |
22,592 |
467.1 |
2.06 |
6,323 |
Qwen3.5-0.8B-LLM |
LLM |
Vanilla |
INT4 AWQ |
- |
1 |
62 |
24.8 |
2,491.8 |
- |
- |
- |
146.3 |
- |
2,293 |
Qwen3.5-0.8B-LLM |
LLM |
Vanilla |
INT4 AWQ |
- |
8 |
441 |
154.4 |
2,856.9 |
- |
- |
- |
663.7 |
- |
3,212 |
Qwen3.5-27B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
1 |
296 |
1,072.4 |
276.4 |
53.8 |
265 |
4,938.9 |
10.5 |
- |
24,017 |
Qwen3.5-27B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
8 |
2,118 |
7,796.9 |
271.6 |
358.3 |
1,896 |
5,292.3 |
42.1 |
- |
26,327 |
Qwen3.5-27B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
1 |
296 |
1,073 |
276.3 |
54.0 |
265 |
4,919 |
18.0 |
2.84 |
25,610 |
Qwen3.5-27B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
8 |
2,118 |
7,808.6 |
271.2 |
358.4 |
1,896 |
5,291 |
63.6 |
2.85 |
33,173 |
Qwen3.5-27B-LLM |
LLM |
Vanilla |
INT4 AWQ |
- |
1 |
62 |
267.5 |
230.9 |
- |
- |
- |
10.5 |
- |
24,005 |
Qwen3.5-27B-LLM |
LLM |
Vanilla |
INT4 AWQ |
- |
8 |
441 |
2,785.1 |
158.4 |
- |
- |
- |
45.0 |
- |
26,257 |
Qwen3.5-2B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
1 |
296 |
126.3 |
2,347.5 |
37.0 |
265 |
7,184.5 |
82.0 |
- |
6,818 |
Qwen3.5-2B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
8 |
2,118 |
672.3 |
3,149.7 |
246.9 |
1,896 |
7,679.3 |
353.7 |
- |
7,376 |
Qwen3.5-2B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
1 |
296 |
124.9 |
2,373.7 |
36.9 |
265 |
7,189.4 |
86.9 |
2.42 |
7,946 |
Qwen3.5-2B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
8 |
2,118 |
672.5 |
3,148.6 |
248.1 |
1,896 |
7,644.3 |
358.1 |
2.42 |
9,311 |
Qwen3.5-2B-LLM |
LLM |
Vanilla |
INT4 AWQ |
- |
1 |
62 |
34.0 |
1,814.8 |
- |
- |
- |
82.0 |
- |
4,178 |
Qwen3.5-2B-LLM |
LLM |
Vanilla |
INT4 AWQ |
- |
8 |
441 |
245.1 |
1,800.2 |
- |
- |
- |
406.7 |
- |
4,934 |
Qwen3.5-35B-A3B |
VLM |
Vanilla |
INT4 GPTQ / FP16 |
COCO |
1 |
296 |
299.9 |
988.6 |
53.4 |
265 |
4,968.9 |
30.1 |
- |
28,865 |
Qwen3.5-35B-A3B |
VLM |
Vanilla |
INT4 GPTQ / FP16 |
COCO |
8 |
2,118 |
1,401.6 |
1,510.8 |
358.8 |
1,896 |
5,284.3 |
128.9 |
- |
26,705 |
Qwen3.5-35B-A3B-LLM |
LLM |
Vanilla |
INT4 GPTQ |
- |
1 |
62 |
116.8 |
528.8 |
- |
- |
- |
31.1 |
- |
35,722 |
Qwen3.5-35B-A3B-LLM |
LLM |
Vanilla |
INT4 GPTQ |
- |
8 |
441 |
523.8 |
842.2 |
- |
- |
- |
134.1 |
- |
36,655 |
Qwen3.5-4B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
1 |
296 |
243.5 |
1,217.6 |
37.0 |
265 |
7,170.8 |
45.7 |
- |
8,450 |
Qwen3.5-4B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
8 |
2,118 |
1,605 |
1,319.4 |
248.7 |
1,896 |
7,624.3 |
170.8 |
- |
9,683 |
Qwen3.5-4B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
1 |
296 |
243.4 |
1,218.2 |
37.0 |
265 |
7,178.7 |
54.8 |
2.53 |
10,083 |
Qwen3.5-4B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
8 |
2,118 |
1,606.5 |
1,318.2 |
248.5 |
1,896 |
7,630.2 |
194.6 |
2.56 |
12,896 |
Qwen3.5-4B-LLM |
LLM |
Vanilla |
INT4 AWQ |
- |
1 |
62 |
64.6 |
956.5 |
- |
- |
- |
45.7 |
- |
6,106 |
Qwen3.5-4B-LLM |
LLM |
Vanilla |
INT4 AWQ |
- |
8 |
441 |
570.6 |
773.2 |
- |
- |
- |
192.0 |
- |
7,555 |
Qwen3.5-9B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
1 |
296 |
353.5 |
838.6 |
53.8 |
265 |
4,938.9 |
28.0 |
- |
12,106 |
Qwen3.5-9B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
8 |
2,118 |
2,355.5 |
899.0 |
358.3 |
1,896 |
5,291.6 |
124.4 |
- |
13,409 |
Qwen3.5-9B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
1 |
296 |
352.2 |
841.8 |
53.6 |
265 |
4,949.3 |
39.8 |
2.79 |
14,526 |
Qwen3.5-9B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
8 |
2,118 |
2,365.1 |
895.4 |
355.0 |
1,896 |
5,341.5 |
165.9 |
2.79 |
17,457 |
Qwen3.5-9B-LLM |
LLM |
Vanilla |
INT4 AWQ |
- |
1 |
62 |
90.1 |
685.7 |
- |
- |
- |
28.0 |
- |
9,959 |
Qwen3.5-9B-LLM |
LLM |
Vanilla |
INT4 AWQ |
- |
8 |
441 |
852.5 |
517.5 |
- |
- |
- |
130.4 |
- |
11,237 |
Qwen3.6-27B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
1 |
296 |
1,072.9 |
276.3 |
53.7 |
265 |
4,944.9 |
17.7 |
2.79 |
25,554 |
Qwen3.6-27B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
8 |
2,118 |
7,793.2 |
271.7 |
358.2 |
1,896 |
5,293.5 |
65.2 |
2.79 |
33,225 |
Jetson Orin NX (16GB)#
Model |
Kind |
Mode |
Precision |
Dataset |
Batch |
Prefill Seq Len |
Prefill Time (ms) |
Prefill (tok/s) |
ViT Time (ms) |
ViT Tok/Run |
ViT (tok/s) |
Decode (tok/s) |
Accept Rate |
GPU Mem (MB) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Qwen2.5-VL-7B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
1 |
376 |
535.5 |
702.6 |
208.5 |
349 |
1,675.2 |
33.0 |
4.76 |
9,161 |
Qwen3-0.6B |
LLM |
Vanilla |
INT4 AWQ |
- |
1 |
61 |
18.9 |
3,237.4 |
- |
- |
- |
117.4 |
- |
1,937 |
Qwen3-1.7B |
LLM |
Vanilla |
INT4 AWQ |
- |
1 |
61 |
35.4 |
1,729.3 |
- |
- |
- |
60.4 |
- |
3,250 |
Qwen3-1.7B |
LLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ |
- |
1 |
61 |
35.4 |
1,728.4 |
- |
- |
- |
73.5 |
3.19 |
3,315 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
INT4 AWQ |
- |
1 |
57 |
74.9 |
763.8 |
- |
- |
- |
31.6 |
- |
4,921 |
Qwen3-8B |
LLM |
Vanilla |
INT4 AWQ |
- |
1 |
61 |
127.2 |
481.2 |
- |
- |
- |
19.0 |
- |
7,880 |
Qwen3-8B |
LLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ |
- |
1 |
61 |
129.1 |
474.2 |
- |
- |
- |
30.7 |
4.18 |
7,999 |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
1 |
292 |
103.1 |
2,836.1 |
82.1 |
265 |
3,234.1 |
60.1 |
- |
4,344 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
1 |
292 |
244.6 |
1,195.9 |
83.7 |
265 |
3,172.1 |
31.2 |
- |
5,903 |
Qwen3-VL-4B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
1 |
292 |
243.0 |
1,203.3 |
83.1 |
265 |
3,195.6 |
54.2 |
4.92 |
6,362 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
1 |
292 |
462.1 |
632.9 |
120.0 |
265 |
2,211.6 |
18.9 |
- |
9,065 |
Qwen3-VL-8B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
1 |
292 |
498.2 |
587.0 |
121.3 |
265 |
2,188 |
28.5 |
3.94 |
9,518 |
Qwen3.5-0.8B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
1 |
296 |
140.5 |
2,110.8 |
26.8 |
265 |
9,891.9 |
88.7 |
- |
2,576 |
Qwen3.5-0.8B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
1 |
296 |
140.2 |
2,114.4 |
26.9 |
265 |
9,861.5 |
74.5 |
2.07 |
3,208 |
Qwen3.5-0.8B-LLM |
LLM |
Vanilla |
INT4 AWQ |
- |
1 |
62 |
37.3 |
1,655.6 |
- |
- |
- |
88.9 |
- |
2,257 |
Qwen3.5-2B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
1 |
296 |
199.6 |
1,484.9 |
78.8 |
265 |
3,367.7 |
48.9 |
- |
4,609 |
Qwen3.5-2B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
1 |
296 |
199.3 |
1,487.2 |
77.6 |
265 |
3,420.9 |
47.7 |
2.41 |
5,698 |
Qwen3.5-2B-LLM |
LLM |
Vanilla |
INT4 AWQ |
- |
1 |
62 |
55.8 |
1,107.6 |
- |
- |
- |
48.5 |
- |
4,145 |
Qwen3.5-4B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
1 |
296 |
424.5 |
698.3 |
79.1 |
265 |
3,355.9 |
26.4 |
- |
6,303 |
Qwen3.5-4B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
1 |
296 |
422.6 |
701.5 |
78.8 |
265 |
3,370 |
29.5 |
2.55 |
7,782 |
Qwen3.5-4B-LLM |
LLM |
Vanilla |
INT4 AWQ |
- |
1 |
62 |
112.5 |
549.2 |
- |
- |
- |
26.5 |
- |
6,113 |
Jetson Orin Nano (8GB)#
Model |
Kind |
Mode |
Precision |
Dataset |
Batch |
Prefill Seq Len |
Prefill Time (ms) |
Prefill (tok/s) |
ViT Time (ms) |
ViT Tok/Run |
ViT (tok/s) |
Decode (tok/s) |
Accept Rate |
GPU Mem (MB) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Qwen3-0.6B |
LLM |
Vanilla |
INT4 AWQ |
- |
1 |
61 |
28.7 |
2,133 |
- |
- |
- |
72.8 |
- |
1,889 |
Qwen3-1.7B |
LLM |
Vanilla |
INT4 AWQ |
- |
1 |
61 |
63.1 |
970.5 |
- |
- |
- |
37.4 |
- |
3,234 |
Qwen3-1.7B |
LLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ |
- |
1 |
61 |
63.2 |
967.8 |
- |
- |
- |
41.2 |
3.19 |
3,328 |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
1 |
292 |
190.9 |
1,532.2 |
151.1 |
265 |
1,756.4 |
36.9 |
- |
4,380 |
Qwen3.5-0.8B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
1 |
296 |
251.9 |
1,176.8 |
49.2 |
265 |
5,400.2 |
55.3 |
- |
2,603 |
Qwen3.5-0.8B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
COCO |
1 |
296 |
251.8 |
1,177.4 |
49.1 |
265 |
5,404.1 |
45.0 |
2.07 |
3,202 |
Qwen3.5-0.8B-LLM |
LLM |
Vanilla |
INT4 AWQ |
- |
1 |
62 |
65.6 |
941.7 |
- |
- |
- |
55.4 |
- |
2,316 |
Qwen3.5-2B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
1 |
296 |
360.4 |
822.6 |
143.2 |
265 |
1,853.8 |
29.9 |
- |
4,621 |
Qwen3.5-2B-LLM |
LLM |
Vanilla |
INT4 AWQ |
- |
1 |
62 |
99.5 |
621.0 |
- |
- |
- |
30.0 |
- |
4,176 |
DGX Spark (GB10)#
Model |
Kind |
Mode |
Precision |
Dataset |
Batch |
Prefill Seq Len |
Prefill Time (ms) |
Prefill (tok/s) |
ViT Time (ms) |
ViT Tok/Run |
ViT (tok/s) |
Decode (tok/s) |
Accept Rate |
GPU Mem (MB) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
NVIDIA-Nemotron-3-Nano-30B-A3B |
LLM |
Vanilla |
NVFP4 |
MTBench |
1 |
66 |
77.6 |
848.7 |
- |
- |
- |
70.9 |
- |
18,537 |
NVIDIA-Nemotron-3-Nano-30B-A3B |
LLM |
Vanilla |
NVFP4 |
MTBench |
8 |
470 |
186.2 |
2,525.9 |
- |
- |
- |
207.9 |
- |
18,713 |
NVIDIA-Nemotron-3-Nano-4B |
LLM |
Vanilla |
NVFP4 |
MTBench |
1 |
66 |
30.2 |
2,180.7 |
- |
- |
- |
59.4 |
- |
3,611 |
NVIDIA-Nemotron-3-Nano-4B |
LLM |
Vanilla |
NVFP4 |
MTBench |
8 |
470 |
107.6 |
4,370.8 |
- |
- |
- |
330.4 |
- |
3,610 |
Qwen3-0.6B |
LLM |
Vanilla |
NVFP4 |
MTBench |
1 |
61 |
6.9 |
8,896.0 |
- |
- |
- |
273.7 |
- |
1,027 |
Qwen3-0.6B |
LLM |
Vanilla |
NVFP4 |
MTBench |
8 |
437 |
15.7 |
27,878.9 |
- |
- |
- |
1,531.3 |
- |
972 |
Qwen3-1.7B |
LLM |
Vanilla |
NVFP4 |
MTBench |
1 |
61 |
11.3 |
5,422.5 |
- |
- |
- |
137.4 |
- |
1,787 |
Qwen3-1.7B |
LLM |
Vanilla |
NVFP4 |
MTBench |
8 |
437 |
27.6 |
15,851.2 |
- |
- |
- |
824.5 |
- |
1,845 |
Qwen3-30B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
MTBench |
1 |
61 |
54.7 |
1,118.6 |
- |
- |
- |
84.9 |
- |
14,052 |
Qwen3-30B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
MTBench |
8 |
437 |
147.3 |
2,966.8 |
- |
- |
- |
232.2 |
- |
13,883 |
Qwen3-30B-A3B |
LLM |
Vanilla |
NVFP4 |
MTBench |
1 |
61 |
65.5 |
934.3 |
- |
- |
- |
80.0 |
- |
17,193 |
Qwen3-30B-A3B |
LLM |
Vanilla |
NVFP4 |
MTBench |
8 |
437 |
154.7 |
2,825.8 |
- |
- |
- |
220.3 |
- |
15,344 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
INT4 AWQ |
MTBench |
1 |
57 |
22.2 |
2,576.2 |
- |
- |
- |
82.8 |
- |
1,854 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
INT4 AWQ |
MTBench |
8 |
409 |
114.4 |
3,571.0 |
- |
- |
- |
389.3 |
- |
1,857 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
NVFP4 |
MTBench |
1 |
57 |
20.4 |
2,805.8 |
- |
- |
- |
71.8 |
- |
3,152 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
NVFP4 |
MTBench |
8 |
409 |
55.6 |
7,345.9 |
- |
- |
- |
435.6 |
- |
3,207 |
Qwen3-8B |
LLM |
Vanilla |
INT4 AWQ |
MTBench |
1 |
61 |
34.7 |
1,765.8 |
- |
- |
- |
47.4 |
- |
3,210 |
Qwen3-8B |
LLM |
Vanilla |
INT4 AWQ |
MTBench |
8 |
437 |
239.6 |
1,824.6 |
- |
- |
- |
252.2 |
- |
3,205 |
Qwen3-8B |
LLM |
Vanilla |
NVFP4 |
MTBench |
1 |
61 |
32.3 |
1,896.8 |
- |
- |
- |
40.6 |
- |
5,371 |
Qwen3-8B |
LLM |
Vanilla |
NVFP4 |
MTBench |
8 |
437 |
87.6 |
4,987.6 |
- |
- |
- |
258.3 |
- |
5,367 |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
1 |
292 |
12.2 |
24,061.0 |
15.3 |
265 |
17,391.3 |
136.8 |
- |
1,891 |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
8 |
2,089 |
67.6 |
30,902.3 |
93.0 |
1,896 |
20,366.6 |
618.5 |
- |
1,893 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
1 |
292 |
48.3 |
6,048.9 |
14.9 |
265 |
17,825.3 |
79.1 |
- |
1,846 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
8 |
2,089 |
311.4 |
6,708.7 |
97.3 |
1,896 |
19,493.2 |
332.4 |
- |
1,902 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
1 |
292 |
27.8 |
10,510.7 |
14.7 |
265 |
18,050.5 |
68.5 |
- |
3,195 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
8 |
2,089 |
149.2 |
13,998.4 |
92.8 |
1,896 |
20,408.2 |
310.2 |
- |
3,192 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
1 |
292 |
87.8 |
3,330.2 |
23.0 |
265 |
11,520.7 |
46.5 |
- |
3,252 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
8 |
2,089 |
618.5 |
3,377.4 |
143.9 |
1,896 |
13,175.2 |
239.5 |
- |
3,193 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
1 |
292 |
40.6 |
7,201.1 |
22.7 |
265 |
11,723.3 |
40.1 |
- |
5,473 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
8 |
2,089 |
245.9 |
8,495.5 |
143.9 |
1,896 |
13,175.2 |
223.0 |
- |
5,281 |
Qwen3.5-0.8B |
LLM |
Vanilla |
NVFP4 |
MTBench |
1 |
62 |
13.7 |
4,523.5 |
- |
- |
- |
212.6 |
- |
1,245 |
Qwen3.5-0.8B |
LLM |
Vanilla |
NVFP4 |
MTBench |
8 |
441 |
36.7 |
12,035.4 |
- |
- |
- |
1,160.9 |
- |
1,252 |
Qwen3.5-0.8B |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
1 |
296 |
37.4 |
7,921.4 |
4.9 |
265 |
54,644.8 |
220.7 |
- |
1,293 |
Qwen3.5-0.8B |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
8 |
2,118 |
106.9 |
19,815.2 |
30.4 |
1,896 |
62,500.0 |
902.1 |
- |
1,296 |
Qwen3.5-27B |
LLM |
Vanilla |
INT4 AWQ |
MTBench |
1 |
62 |
134.1 |
460.7 |
- |
- |
- |
14.6 |
- |
8,681 |
Qwen3.5-27B |
LLM |
Vanilla |
INT4 AWQ |
MTBench |
8 |
441 |
1,183.8 |
372.6 |
- |
- |
- |
46.1 |
- |
8,871 |
Qwen3.5-27B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
1 |
296 |
513.3 |
577.6 |
22.8 |
265 |
11,614.4 |
6.6 |
- |
8,820 |
Qwen3.5-27B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
8 |
2,118 |
3,088.9 |
685.5 |
141.8 |
1,896 |
13,369.0 |
69.3 |
- |
8,930 |
Qwen3.5-27B |
LLM |
Vanilla |
NVFP4 |
MTBench |
1 |
62 |
110.8 |
557.2 |
- |
- |
- |
12.9 |
- |
16,079 |
Qwen3.5-27B |
LLM |
Vanilla |
NVFP4 |
MTBench |
8 |
441 |
381.3 |
1,156.9 |
- |
- |
- |
78.4 |
- |
16,017 |
Qwen3.5-27B |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
1 |
296 |
201.2 |
1,473.8 |
20.8 |
265 |
12,755.1 |
12.8 |
- |
16,124 |
Qwen3.5-27B |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
8 |
2,118 |
1,191.1 |
1,777.8 |
138.9 |
1,896 |
13,661.2 |
66.2 |
- |
16,121 |
Qwen3.5-2B |
LLM |
Vanilla |
NVFP4 |
MTBench |
1 |
62 |
18.4 |
3,364.8 |
- |
- |
- |
115.0 |
- |
2,207 |
Qwen3.5-2B |
LLM |
Vanilla |
NVFP4 |
MTBench |
8 |
441 |
48.3 |
9,124.0 |
- |
- |
- |
656.7 |
- |
2,204 |
Qwen3.5-2B |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
1 |
296 |
44.1 |
6,730.0 |
13.7 |
265 |
19,455.3 |
112.1 |
- |
2,251 |
Qwen3.5-2B |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
8 |
2,118 |
131.4 |
16,114.6 |
86.9 |
1,896 |
21,834.1 |
456.9 |
- |
2,254 |
Qwen3.5-35B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
MTBench |
1 |
62 |
69.8 |
884.9 |
- |
- |
- |
47.3 |
- |
15,349 |
Qwen3.5-35B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
MTBench |
8 |
441 |
203.5 |
2,167.4 |
- |
- |
- |
191.7 |
- |
15,341 |
Qwen3.5-35B-A3B |
VLM |
Vanilla |
INT4 GPTQ / FP16 |
COCO |
1 |
296 |
147.3 |
2,013.2 |
20.9 |
265 |
12,706.5 |
45.6 |
- |
15,666 |
Qwen3.5-35B-A3B |
VLM |
Vanilla |
INT4 GPTQ / FP16 |
COCO |
8 |
2,118 |
494.0 |
4,286.6 |
137.3 |
1,896 |
13,812.2 |
174.7 |
- |
15,470 |
Qwen3.5-4B |
LLM |
Vanilla |
INT4 AWQ |
MTBench |
1 |
62 |
35.3 |
1,749.5 |
- |
- |
- |
67.9 |
- |
1,694 |
Qwen3.5-4B |
LLM |
Vanilla |
INT4 AWQ |
MTBench |
8 |
441 |
192.4 |
2,292.5 |
- |
- |
- |
311.6 |
- |
1,755 |
Qwen3.5-4B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
1 |
296 |
100.5 |
2,949.1 |
14.7 |
265 |
18,083.2 |
67.5 |
- |
1,737 |
Qwen3.5-4B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
8 |
2,118 |
554.0 |
3,822.1 |
115.5 |
1,896 |
16,420.4 |
258.9 |
- |
1,742 |
Qwen3.5-4B |
LLM |
Vanilla |
NVFP4 |
MTBench |
1 |
62 |
31.2 |
1,980.0 |
- |
- |
- |
59.2 |
- |
3,587 |
Qwen3.5-4B |
LLM |
Vanilla |
NVFP4 |
MTBench |
8 |
441 |
101.3 |
4,356.8 |
- |
- |
- |
341.5 |
- |
3,647 |
Qwen3.5-4B |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
1 |
296 |
67.9 |
4,368.0 |
13.3 |
265 |
19,960.1 |
63.0 |
- |
3,692 |
Qwen3.5-4B |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
8 |
2,118 |
295.1 |
7,176.8 |
92.6 |
1,896 |
20,491.8 |
240.4 |
- |
3,691 |
Qwen3.5-9B |
LLM |
Vanilla |
INT4 AWQ |
MTBench |
1 |
62 |
49.9 |
1,238.6 |
- |
- |
- |
40.1 |
- |
2,855 |
Qwen3.5-9B |
LLM |
Vanilla |
INT4 AWQ |
MTBench |
8 |
441 |
341.7 |
1,291.0 |
- |
- |
- |
206.5 |
- |
2,859 |
Qwen3.5-9B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
1 |
296 |
145.7 |
2,035.3 |
24.0 |
265 |
11,086.5 |
34.8 |
- |
2,902 |
Qwen3.5-9B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
COCO |
8 |
2,118 |
914.5 |
2,315.6 |
138.2 |
1,896 |
13,717.4 |
197.8 |
- |
2,961 |
Qwen3.5-9B |
LLM |
Vanilla |
NVFP4 |
MTBench |
1 |
62 |
44.6 |
1,384.6 |
- |
- |
- |
35.8 |
- |
6,112 |
Qwen3.5-9B |
LLM |
Vanilla |
NVFP4 |
MTBench |
8 |
441 |
131.6 |
3,352.0 |
- |
- |
- |
214.5 |
- |
6,111 |
Qwen3.5-9B |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
1 |
296 |
82.7 |
3,585.7 |
20.1 |
265 |
13,227.5 |
35.3 |
- |
6,159 |
Qwen3.5-9B |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
8 |
2,118 |
396.1 |
5,345.9 |
139.8 |
1,896 |
13,568.5 |
180.5 |
- |
6,157 |
Qwen3.6-35B-A3B |
LLM |
Vanilla |
NVFP4 |
MTBench |
1 |
62 |
77.3 |
799.3 |
- |
- |
- |
72.1 |
- |
19,464 |
Qwen3.6-35B-A3B |
LLM |
Vanilla |
NVFP4 |
MTBench |
8 |
441 |
204.8 |
2,153.7 |
- |
- |
- |
211.4 |
- |
18,060 |
Qwen3.6-35B-A3B |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
1 |
296 |
165.0 |
1,797.1 |
18.5 |
265 |
14,326.6 |
71.3 |
- |
17,896 |
Qwen3.6-35B-A3B |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
8 |
2,118 |
491.3 |
4,310.2 |
134.6 |
1,896 |
14,084.5 |
223.0 |
- |
18,567 |
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
1 |
1,699 |
275.6 |
6,163.7 |
157.0 |
1,664 |
10,593.2 |
64.9 |
- |
19,790 |
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 |
VLM |
Vanilla |
NVFP4 / FP16 |
COCO |
8 |
12,136 |
1,718.3 |
7,062.6 |
1,120.1 |
11,886 |
10,615.7 |
155.0 |
- |
18,435 |
v0.9.0 Collection Method#
Engines were built from exported v0.9.0 ONNX artifacts using the build limits in Export and Build Specs.
Runtime throughput was collected with
llm_inference --warmup 10 --dump-profile --profile-output-file <profile.json>using benchmark JSON inputs for each model family.Synthetic component timing was collected with
llm_bench --warmup 3 --iterations 10; prefill uses--input-len 2048and decode uses--past-kv-len 2048.Jetson AGX Thor and DGX Spark runs include the supported NVFP4 and INT4 entries. Jetson AGX Orin, Orin NX, and Orin Nano run the externalized INT4 entries supported by each memory target.
v0.8.0 Results#
SDK Version: TensorRT Edge-LLM 0.8.0 | JetPack: 7.2 | Source: v0.8.0 release benchmark outputs | Devices: Jetson AGX Thor, Jetson AGX Orin 64GB, Jetson Orin NX 16GB, Jetson Orin Nano 8GB
Limitation: v0.8.0 has a uniform performance regression across the benchmarked release devices. This regression is fixed in v0.9.0, so use v0.9.0 or later for current performance expectations.
llm_bench Commands#
The v0.8.0 release benchmarks use llm_bench for synthetic component timing. The release run used --warmup=2, --iterations=10, and --profile; replace paths and lengths with the engine directory and shape listed in the table.
# Prefill throughput
./build/bin/llm_bench \
--engine-dir <llm_engine_dir> \
--mode prefill \
--batch-size <batch_size> \
--input-len <input_len> \
--warmup 2 \
--iterations 10 \
--profile
# Decode throughput
./build/bin/llm_bench \
--engine-dir <llm_engine_dir> \
--mode decode \
--batch-size <batch_size> \
--past-kv-len <past_kv_len> \
--warmup 2 \
--iterations 10 \
--profile
# Speculative decoding component timing
./build/bin/llm_bench --engine-dir <engine_dir> --mode spec_draft_prefill --batch-size <batch_size> --input-len <input_len> --warmup 2 --iterations 10 --profile
./build/bin/llm_bench --engine-dir <engine_dir> --mode spec_draft_proposal --batch-size <batch_size> --draft-tree-size <draft_tree_size> --past-kv-len <past_kv_len> --warmup 2 --iterations 10 --profile
./build/bin/llm_bench --engine-dir <engine_dir> --mode spec_verify --batch-size <batch_size> --verify-tree-size <verify_tree_size> --past-kv-len <past_kv_len> --warmup 2 --iterations 10 --profile
# Visual encoder timing
./build/bin/llm_bench --engine-dir <visual_engine_dir> --mode visual --image-size <height>x<width> --warmup 2 --iterations 10 --profile
llm_bench Prefill Performance (Jetson AGX Thor Only)#
These are the parsed synthetic llm_bench_prefill_* results in the v0.8.0 release benchmark outputs. Only the Jetson AGX Thor data contains parsed llm_bench_prefill_e2e_time_ms and llm_bench_prefill_tokens_per_sec values. The AGX Orin, Orin NX, and Orin Nano data includes runtime prefill metrics in the dashboard below, but does not include those synthetic llm_bench prefill e2e/tok/s fields.
Platform |
Model |
Kind |
Mode |
Precision |
Batch |
Input Len |
Prefill E2E (ms) |
Prefill (tok/s) |
|---|---|---|---|---|---|---|---|---|
Jetson AGX Thor |
NVIDIA-Nemotron-3-Nano-30B-A3B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
228.1 |
8,978.9 |
Jetson AGX Thor |
NVIDIA-Nemotron-3-Nano-30B-A3B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
1,385.7 |
1,478.0 |
Jetson AGX Thor |
NVIDIA-Nemotron-3-Nano-4B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
1,152.7 |
1,776.7 |
Jetson AGX Thor |
NVIDIA-Nemotron-3-Nano-4B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
5,347.0 |
383.0 |
Jetson AGX Thor |
Qwen3-0.6B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
22.4 |
91,469.3 |
Jetson AGX Thor |
Qwen3-0.6B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
248.5 |
8,241.0 |
Jetson AGX Thor |
Qwen3-1.7B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
32.3 |
63,441.6 |
Jetson AGX Thor |
Qwen3-1.7B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
356.8 |
5,739.6 |
Jetson AGX Thor |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
67.1 |
30,539.5 |
Jetson AGX Thor |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
823.8 |
2,486.0 |
Jetson AGX Thor |
Qwen3-8B |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
111.0 |
18,444.1 |
Jetson AGX Thor |
Qwen3-8B |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
1,245.9 |
1,643.8 |
Jetson AGX Thor |
Qwen3.5-0.8B-LLM |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
36.6 |
55,904.4 |
Jetson AGX Thor |
Qwen3.5-0.8B-LLM |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
349.0 |
5,868.1 |
Jetson AGX Thor |
Qwen3.5-27B-LLM |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
463.3 |
4,420.5 |
Jetson AGX Thor |
Qwen3.5-27B-LLM |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
4,268.6 |
479.8 |
Jetson AGX Thor |
Qwen3.5-2B-LLM |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
46.1 |
44,458.0 |
Jetson AGX Thor |
Qwen3.5-2B-LLM |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
435.6 |
4,701.8 |
Jetson AGX Thor |
Qwen3.5-4B-LLM |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
101.2 |
20,237.3 |
Jetson AGX Thor |
Qwen3.5-4B-LLM |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
997.1 |
2,053.9 |
Jetson AGX Thor |
Qwen3.5-9B-LLM |
LLM |
Vanilla |
NVFP4 |
1 |
2,048 |
138.4 |
14,795.8 |
Jetson AGX Thor |
Qwen3.5-9B-LLM |
LLM |
Vanilla |
NVFP4 |
8 |
2,048 |
1,411.1 |
1,451.3 |
Jetson AGX Thor |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
32.6 |
62,763.4 |
Jetson AGX Thor |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
358.0 |
5,721.4 |
Jetson AGX Thor |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
68.5 |
29,908.3 |
Jetson AGX Thor |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
833.3 |
2,457.8 |
Jetson AGX Thor |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
112.2 |
18,254.6 |
Jetson AGX Thor |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
1,239.3 |
1,652.6 |
Jetson AGX Thor |
Qwen3.5-0.8B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
36.5 |
56,042.9 |
Jetson AGX Thor |
Qwen3.5-0.8B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
349.2 |
5,864.1 |
Jetson AGX Thor |
Qwen3.5-27B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
459.2 |
4,460.3 |
Jetson AGX Thor |
Qwen3.5-27B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
4,201.7 |
487.4 |
Jetson AGX Thor |
Qwen3.5-2B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
46.0 |
44,568.0 |
Jetson AGX Thor |
Qwen3.5-2B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
438.0 |
4,676.0 |
Jetson AGX Thor |
Qwen3.5-4B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
101.2 |
20,235.5 |
Jetson AGX Thor |
Qwen3.5-4B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
996.5 |
2,055.2 |
Jetson AGX Thor |
Qwen3.5-9B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
138.7 |
14,766.6 |
Jetson AGX Thor |
Qwen3.5-9B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
1,399.9 |
1,462.9 |
Jetson AGX Thor |
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
2,048 |
223.6 |
9,157.3 |
Jetson AGX Thor |
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
2,048 |
1,355.9 |
1,510.4 |
Jetson AGX Thor |
Qwen3-1.7B |
LLM |
EAGLE3 |
NVFP4 / NVFP4 |
1 |
2,048 |
30.6 |
66,929.4 |
Jetson AGX Thor |
Qwen3-1.7B |
LLM |
EAGLE3 |
NVFP4 / NVFP4 |
8 |
2,048 |
356.9 |
5,738.5 |
Jetson AGX Thor |
Qwen3-8B |
LLM |
EAGLE3 |
NVFP4 / NVFP4 |
1 |
2,048 |
109.2 |
18,754.4 |
Jetson AGX Thor |
Qwen3-8B |
LLM |
EAGLE3 |
NVFP4 / NVFP4 |
8 |
2,048 |
1,241.7 |
1,649.4 |
Jetson AGX Thor |
Qwen2.5-VL-7B-Instruct |
VLM |
EAGLE3 |
NVFP4 / NVFP4 / FP16 |
1 |
2,048 |
81.8 |
25,038.0 |
Jetson AGX Thor |
Qwen2.5-VL-7B-Instruct |
VLM |
EAGLE3 |
NVFP4 / NVFP4 / FP16 |
8 |
2,048 |
984.0 |
2,081.3 |
Jetson AGX Thor |
Qwen3-VL-4B-Instruct |
VLM |
EAGLE3 |
NVFP4 / NVFP4 / FP16 |
1 |
2,048 |
64.6 |
31,680.9 |
Jetson AGX Thor |
Qwen3-VL-4B-Instruct |
VLM |
EAGLE3 |
NVFP4 / NVFP4 / FP16 |
8 |
2,048 |
833.8 |
2,456.2 |
Jetson AGX Thor |
Qwen3.5-0.8B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
2,048 |
34.9 |
58,600.2 |
Jetson AGX Thor |
Qwen3.5-0.8B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
2,048 |
348.8 |
5,872.0 |
Jetson AGX Thor |
Qwen3.5-27B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
2,048 |
454.5 |
4,505.6 |
Jetson AGX Thor |
Qwen3.5-27B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
2,048 |
4,209.1 |
486.6 |
Jetson AGX Thor |
Qwen3.5-2B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
2,048 |
43.0 |
47,649.3 |
Jetson AGX Thor |
Qwen3.5-2B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
2,048 |
433.9 |
4,719.7 |
Jetson AGX Thor |
Qwen3.5-4B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
2,048 |
99.2 |
20,648.3 |
Jetson AGX Thor |
Qwen3.5-4B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
2,048 |
994.7 |
2,058.9 |
Jetson AGX Thor |
Qwen3.5-9B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
2,048 |
134.2 |
15,265.9 |
Jetson AGX Thor |
Qwen3.5-9B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
2,048 |
1,399.6 |
1,463.3 |
Runtime Performance Dashboard#
All v0.8.0 runtime entries below were benchmarked under JetPack 7.2 and are split by device.
Jetson AGX Thor#
Model |
Kind |
Mode |
Precision |
Batch |
Runtime Prefill (ms) |
Runtime Prefill Tok/Run |
Runtime Prefill (tok/s) |
ViT (ms) |
ViT Tok/Run |
ViT (tok/s) |
Generation (tok/s) |
Accept Rate |
GPU Mem (MB) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
NVIDIA-Nemotron-3-Nano-30B-A3B |
LLM |
Vanilla |
NVFP4 |
1 |
104.8 |
383 |
3,653.0 |
- |
- |
- |
72.5 |
- |
19,987 |
NVIDIA-Nemotron-3-Nano-30B-A3B |
LLM |
Vanilla |
NVFP4 |
8 |
508.9 |
3,062 |
6,016.5 |
- |
- |
- |
180.0 |
- |
19,975 |
NVIDIA-Nemotron-3-Nano-4B |
LLM |
Vanilla |
NVFP4 |
1 |
120.8 |
383 |
3,169.5 |
- |
- |
- |
66.4 |
- |
3,538 |
NVIDIA-Nemotron-3-Nano-4B |
LLM |
Vanilla |
NVFP4 |
8 |
933.8 |
3,062 |
3,279.1 |
- |
- |
- |
302.9 |
- |
3,548 |
Qwen3-0.6B |
LLM |
Vanilla |
INT4 AWQ |
1 |
22.7 |
370 |
16,296.0 |
- |
- |
- |
194.0 |
- |
773 |
Qwen3-0.6B |
LLM |
Vanilla |
INT4 AWQ |
8 |
238.9 |
2,959 |
12,383.9 |
- |
- |
- |
329.6 |
- |
875 |
Qwen3-0.6B |
LLM |
Vanilla |
NVFP4 |
1 |
13.4 |
370 |
27,533.2 |
- |
- |
- |
192.5 |
- |
957 |
Qwen3-0.6B |
LLM |
Vanilla |
NVFP4 |
8 |
100.5 |
2,959 |
29,449.6 |
- |
- |
- |
359.8 |
- |
998 |
Qwen3-1.7B |
LLM |
Vanilla |
INT4 AWQ |
1 |
49.8 |
370 |
7,423.1 |
- |
- |
- |
119.7 |
- |
1,067 |
Qwen3-1.7B |
LLM |
Vanilla |
INT4 AWQ |
8 |
589.9 |
2,959 |
5,015.4 |
- |
- |
- |
276.6 |
- |
1,066 |
Qwen3-1.7B |
LLM |
Vanilla |
NVFP4 |
1 |
18.4 |
370 |
20,067.5 |
- |
- |
- |
118.9 |
- |
1,884 |
Qwen3-1.7B |
LLM |
Vanilla |
NVFP4 |
8 |
142.9 |
2,959 |
20,704.1 |
- |
- |
- |
302.4 |
- |
1,782 |
Qwen3-30B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
1 |
140.2 |
370 |
2,638.4 |
- |
- |
- |
75.1 |
- |
14,305 |
Qwen3-30B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
8 |
1,461.4 |
2,959 |
2,024.6 |
- |
- |
- |
161.7 |
- |
14,313 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
INT4 AWQ |
1 |
111.9 |
364 |
3,251.6 |
- |
- |
- |
66.2 |
- |
1,846 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
INT4 AWQ |
8 |
1,413.0 |
2,911 |
2,060.0 |
- |
- |
- |
173.7 |
- |
1,813 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
NVFP4 |
1 |
31.5 |
364 |
11,568.8 |
- |
- |
- |
67.1 |
- |
3,168 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
NVFP4 |
8 |
283.9 |
2,911 |
10,251.5 |
- |
- |
- |
216.3 |
- |
3,148 |
Qwen3-8B |
LLM |
Vanilla |
INT4 AWQ |
1 |
200.3 |
370 |
1,846.7 |
- |
- |
- |
43.7 |
- |
3,195 |
Qwen3-8B |
LLM |
Vanilla |
INT4 AWQ |
8 |
2,546.9 |
2,959 |
1,161.7 |
- |
- |
- |
126.8 |
- |
3,260 |
Qwen3-8B |
LLM |
Vanilla |
NVFP4 |
1 |
42.8 |
370 |
8,640.5 |
- |
- |
- |
42.3 |
- |
5,356 |
Qwen3-8B |
LLM |
Vanilla |
NVFP4 |
8 |
426.8 |
2,959 |
6,933.1 |
- |
- |
- |
157.4 |
- |
5,376 |
Qwen3.5-0.8B-LLM |
LLM |
Vanilla |
INT4 AWQ |
1 |
30.7 |
377 |
12,271.7 |
- |
- |
- |
214.5 |
- |
951 |
Qwen3.5-0.8B-LLM |
LLM |
Vanilla |
INT4 AWQ |
8 |
370.8 |
3,013 |
8,126.8 |
- |
- |
- |
717.8 |
- |
962 |
Qwen3.5-0.8B-LLM |
LLM |
Vanilla |
NVFP4 |
1 |
13.8 |
377 |
27,269.9 |
- |
- |
- |
216.2 |
- |
1,176 |
Qwen3.5-0.8B-LLM |
LLM |
Vanilla |
NVFP4 |
8 |
114.6 |
3,013 |
26,298.4 |
- |
- |
- |
926.5 |
- |
1,172 |
Qwen3.5-27B-LLM |
LLM |
Vanilla |
INT4 AWQ |
1 |
713.9 |
377 |
527.6 |
- |
- |
- |
15.4 |
- |
8,927 |
Qwen3.5-27B-LLM |
LLM |
Vanilla |
INT4 AWQ |
8 |
10,102.2 |
3,013 |
298.3 |
- |
- |
- |
54.6 |
- |
8,940 |
Qwen3.5-27B-LLM |
LLM |
Vanilla |
NVFP4 |
1 |
130.5 |
377 |
2,887.1 |
- |
- |
- |
14.3 |
- |
15,990 |
Qwen3.5-27B-LLM |
LLM |
Vanilla |
NVFP4 |
8 |
1,356.3 |
3,013 |
2,221.6 |
- |
- |
- |
72.9 |
- |
16,044 |
Qwen3.5-2B-LLM |
LLM |
Vanilla |
INT4 AWQ |
1 |
56.6 |
377 |
6,650.8 |
- |
- |
- |
118.8 |
- |
1,443 |
Qwen3.5-2B-LLM |
LLM |
Vanilla |
INT4 AWQ |
8 |
690.0 |
3,013 |
4,366.8 |
- |
- |
- |
459.4 |
- |
1,446 |
Qwen3.5-2B-LLM |
LLM |
Vanilla |
NVFP4 |
1 |
18.8 |
377 |
19,993.5 |
- |
- |
- |
118.8 |
- |
2,131 |
Qwen3.5-2B-LLM |
LLM |
Vanilla |
NVFP4 |
8 |
148.7 |
3,013 |
20,261.3 |
- |
- |
- |
601.0 |
- |
2,138 |
Qwen3.5-35B-A3B-LLM |
LLM |
Vanilla |
INT4 GPTQ |
1 |
118.7 |
377 |
3,173.7 |
- |
- |
- |
46.5 |
- |
15,847 |
Qwen3.5-35B-A3B-LLM |
LLM |
Vanilla |
INT4 GPTQ |
8 |
973.7 |
3,013 |
3,094.7 |
- |
- |
- |
175.0 |
- |
15,840 |
Qwen3.5-4B-LLM |
LLM |
Vanilla |
INT4 AWQ |
1 |
128.5 |
377 |
2,931.7 |
- |
- |
- |
64.9 |
- |
1,707 |
Qwen3.5-4B-LLM |
LLM |
Vanilla |
INT4 AWQ |
8 |
1,660.6 |
3,013 |
1,814.5 |
- |
- |
- |
221.4 |
- |
1,702 |
Qwen3.5-4B-LLM |
LLM |
Vanilla |
NVFP4 |
1 |
34.4 |
377 |
10,939.3 |
- |
- |
- |
64.8 |
- |
3,567 |
Qwen3.5-4B-LLM |
LLM |
Vanilla |
NVFP4 |
8 |
305.0 |
3,013 |
9,880.3 |
- |
- |
- |
303.0 |
- |
3,604 |
Qwen3.5-9B-LLM |
LLM |
Vanilla |
INT4 AWQ |
1 |
219.5 |
377 |
1,715.8 |
- |
- |
- |
41.2 |
- |
2,863 |
Qwen3.5-9B-LLM |
LLM |
Vanilla |
INT4 AWQ |
8 |
3,031.4 |
3,013 |
994.0 |
- |
- |
- |
144.6 |
- |
2,866 |
Qwen3.5-9B-LLM |
LLM |
Vanilla |
NVFP4 |
1 |
46.4 |
377 |
8,112.3 |
- |
- |
- |
39.4 |
- |
6,110 |
Qwen3.5-9B-LLM |
LLM |
Vanilla |
NVFP4 |
8 |
449.7 |
3,013 |
6,700.4 |
- |
- |
- |
181.2 |
- |
6,108 |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
38.2 |
283 |
7,399.3 |
12.6 |
263 |
20,790.0 |
119.0 |
- |
1,422 |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
321.0 |
2,196 |
6,840.7 |
85.9 |
2,036 |
23,696.7 |
273.6 |
- |
1,409 |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
14.8 |
283 |
19,152.2 |
12.6 |
262 |
20,790.0 |
117.8 |
- |
1,867 |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
92.1 |
2,196 |
23,828.6 |
85.6 |
2,039 |
23,809.5 |
309.2 |
- |
1,910 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
89.2 |
283 |
3,169.7 |
13.0 |
262 |
20,161.3 |
67.0 |
- |
1,899 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
775.3 |
2,196 |
2,832.1 |
85.8 |
2,039 |
23,753.0 |
164.2 |
- |
1,914 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
28.4 |
283 |
9,963.3 |
12.7 |
262 |
20,746.9 |
66.6 |
- |
3,228 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
186.2 |
2,196 |
11,791.3 |
85.5 |
2,036 |
23,809.5 |
181.1 |
- |
3,242 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
155.8 |
283 |
1,813.9 |
17.3 |
263 |
15,174.5 |
44.1 |
- |
3,283 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
1,354.5 |
2,196 |
1,621.0 |
119.6 |
2,037 |
17,035.8 |
127.3 |
- |
3,243 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
38.4 |
283 |
7,354.1 |
17.3 |
262 |
15,151.5 |
41.9 |
- |
5,484 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
260.7 |
2,196 |
8,422.7 |
119.5 |
2,037 |
17,035.8 |
150.5 |
- |
5,482 |
Qwen3.5-0.8B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
21.4 |
287 |
13,382.7 |
4.3 |
262 |
60,975.6 |
209.8 |
- |
1,045 |
Qwen3.5-0.8B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
191.0 |
2,227 |
11,660.9 |
30.6 |
2,041 |
66,666.7 |
593.7 |
- |
1,085 |
Qwen3.5-0.8B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
10.0 |
287 |
28,729.9 |
4.3 |
262 |
60,975.6 |
210.9 |
- |
1,260 |
Qwen3.5-0.8B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
63.6 |
2,227 |
34,989.3 |
30.6 |
2,042 |
66,666.7 |
758.0 |
- |
1,268 |
Qwen3.5-27B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
558.3 |
287 |
513.4 |
16.2 |
263 |
16,181.2 |
15.5 |
- |
9,128 |
Qwen3.5-27B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
5,186.5 |
2,227 |
429.4 |
114.6 |
2,039 |
17,793.6 |
54.7 |
- |
9,011 |
Qwen3.5-27B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
103.6 |
287 |
2,765.5 |
16.1 |
262 |
16,286.6 |
14.3 |
- |
16,087 |
Qwen3.5-27B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
744.7 |
2,227 |
2,990.2 |
114.4 |
2,040 |
17,825.3 |
74.3 |
- |
16,119 |
Qwen3.5-2B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
41.8 |
287 |
6,853.9 |
11.7 |
262 |
22,321.4 |
115.9 |
- |
1,562 |
Qwen3.5-2B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
354.2 |
2,227 |
6,286.8 |
83.3 |
2,037 |
24,449.9 |
419.2 |
- |
1,639 |
Qwen3.5-2B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
14.3 |
287 |
20,005.7 |
11.7 |
262 |
22,471.9 |
117.7 |
- |
2,213 |
Qwen3.5-2B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
79.1 |
2,227 |
28,157.1 |
82.1 |
2,037 |
24,813.9 |
450.7 |
- |
2,228 |
Qwen3.5-35B-A3B |
VLM |
Vanilla |
INT4 GPTQ / FP16 |
1 |
109.8 |
287 |
2,609.4 |
16.0 |
262 |
16,366.6 |
46.2 |
- |
15,942 |
Qwen3.5-35B-A3B |
VLM |
Vanilla |
INT4 GPTQ / FP16 |
8 |
522.2 |
2,227 |
4,264.3 |
113.4 |
2,040 |
17,985.6 |
181.6 |
- |
15,944 |
Qwen3.5-4B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
97.2 |
287 |
2,948.2 |
12.0 |
262 |
21,881.8 |
65.0 |
- |
1,788 |
Qwen3.5-4B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
856.1 |
2,227 |
2,601.2 |
82.7 |
2,036 |
24,630.5 |
218.1 |
- |
1,787 |
Qwen3.5-4B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
26.6 |
287 |
10,781.8 |
11.7 |
262 |
22,371.4 |
64.4 |
- |
3,678 |
Qwen3.5-4B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
166.8 |
2,227 |
13,348.1 |
82.0 |
2,040 |
24,875.6 |
287.2 |
- |
3,669 |
Qwen3.5-9B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
170.7 |
287 |
1,678.6 |
16.1 |
262 |
16,286.6 |
41.1 |
- |
2,953 |
Qwen3.5-9B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
1,560.5 |
2,227 |
1,427.0 |
113.2 |
2,037 |
17,985.6 |
145.7 |
- |
2,953 |
Qwen3.5-9B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
38.2 |
287 |
7,512.9 |
16.0 |
262 |
16,366.6 |
39.4 |
- |
6,178 |
Qwen3.5-9B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
243.6 |
2,227 |
9,140.3 |
113.2 |
2,040 |
18,018.0 |
183.2 |
- |
6,207 |
Qwen3.6-35B-A3B |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
144.8 |
287 |
1,979.8 |
16.1 |
262 |
16,313.2 |
68.5 |
- |
21,607 |
Qwen3.6-35B-A3B |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
350.8 |
2,227 |
6,347.9 |
113.2 |
2,039 |
18,018.0 |
199.3 |
- |
21,569 |
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 |
VLM |
Vanilla |
NVFP4 / FP16 |
1 |
209.7 |
1,663 |
7,932.3 |
126.0 |
1,634 |
12,970.2 |
67.5 |
- |
20,259 |
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 |
VLM |
Vanilla |
NVFP4 / FP16 |
8 |
1,182.8 |
12,922 |
10,925.0 |
982.5 |
12,694 |
12,919.9 |
106.9 |
- |
20,257 |
Qwen3-1.7B |
LLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ |
1 |
49.5 |
370 |
7,464.8 |
- |
- |
- |
182.9 |
3.89 |
1,067 |
Qwen3-1.7B |
LLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ |
8 |
590.6 |
2,959 |
5,010.0 |
- |
- |
- |
296.5 |
3.87 |
1,105 |
Qwen3-1.7B |
LLM |
EAGLE3 |
NVFP4 / NVFP4 |
1 |
16.0 |
370 |
23,065.1 |
- |
- |
- |
284.9 |
3.84 |
1,345 |
Qwen3-1.7B |
LLM |
EAGLE3 |
NVFP4 / NVFP4 |
8 |
141.6 |
2,959 |
20,900.3 |
- |
- |
- |
618.0 |
3.81 |
1,371 |
Qwen3-8B |
LLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ |
1 |
199.9 |
370 |
1,849.8 |
- |
- |
- |
70.0 |
4.15 |
3,156 |
Qwen3-8B |
LLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ |
8 |
2,550.1 |
2,959 |
1,160.3 |
- |
- |
- |
95.7 |
4.11 |
3,174 |
Qwen3-8B |
LLM |
EAGLE3 |
NVFP4 / NVFP4 |
1 |
39.3 |
370 |
9,421.5 |
- |
- |
- |
135.2 |
4.06 |
4,499 |
Qwen3-8B |
LLM |
EAGLE3 |
NVFP4 / NVFP4 |
8 |
428.2 |
2,959 |
6,910.2 |
- |
- |
- |
341.0 |
4.05 |
4,550 |
Qwen2.5-VL-7B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
1 |
180.9 |
376 |
2,076.4 |
52.5 |
344 |
6,561.7 |
84.4 |
5.15 |
3,406 |
Qwen2.5-VL-7B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
8 |
1,613.3 |
2,919 |
1,809.0 |
429.4 |
2,675 |
6,230.5 |
116.7 |
5.07 |
3,371 |
Qwen2.5-VL-7B-Instruct |
VLM |
EAGLE3 |
NVFP4 / NVFP4 / FP16 |
1 |
28.6 |
376 |
13,119.4 |
52.5 |
344 |
6,557.4 |
189.6 |
5.09 |
4,308 |
Qwen2.5-VL-7B-Instruct |
VLM |
EAGLE3 |
NVFP4 / NVFP4 / FP16 |
8 |
258.9 |
2,919 |
11,271.9 |
431.0 |
2,675 |
6,207.3 |
383.8 |
4.90 |
4,319 |
Qwen3-VL-4B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
1 |
89.0 |
283 |
3,174.8 |
12.9 |
262 |
20,242.9 |
126.5 |
5.02 |
1,900 |
Qwen3-VL-4B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
8 |
774.9 |
2,196 |
2,833.6 |
85.8 |
2,038 |
23,753.0 |
187.9 |
4.96 |
1,894 |
Qwen3-VL-4B-Instruct |
VLM |
EAGLE3 |
NVFP4 / NVFP4 / FP16 |
1 |
26.2 |
283 |
10,789.2 |
12.6 |
262 |
20,833.3 |
199.7 |
4.86 |
2,685 |
Qwen3-VL-4B-Instruct |
VLM |
EAGLE3 |
NVFP4 / NVFP4 / FP16 |
8 |
186.4 |
2,196 |
11,778.3 |
85.4 |
2,039 |
23,866.3 |
418.8 |
4.87 |
2,700 |
Qwen3-VL-8B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
1 |
156.2 |
283 |
1,809.5 |
17.3 |
262 |
15,151.5 |
48.2 |
2.85 |
3,252 |
Qwen3-VL-8B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
8 |
1,354.6 |
2,196 |
1,621.0 |
120.4 |
2,037 |
16,920.5 |
65.7 |
2.82 |
3,251 |
Qwen3.5-0.8B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
1 |
21.2 |
287 |
13,492.0 |
4.3 |
262 |
60,606.1 |
200.3 |
2.18 |
1,179 |
Qwen3.5-0.8B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
8 |
234.5 |
2,227 |
9,496.0 |
30.7 |
2,032 |
66,225.2 |
517.8 |
2.17 |
1,139 |
Qwen3.5-0.8B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
8.4 |
287 |
34,076.3 |
4.3 |
261 |
60,241.0 |
365.6 |
2.19 |
1,069 |
Qwen3.5-0.8B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
59.2 |
2,227 |
37,589.6 |
30.6 |
2,042 |
66,666.7 |
979.9 |
2.15 |
1,191 |
Qwen3.5-27B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
1 |
558.0 |
287 |
513.6 |
16.2 |
262 |
16,233.8 |
27.9 |
2.89 |
9,022 |
Qwen3.5-27B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
8 |
5,188.0 |
2,227 |
429.2 |
114.3 |
2,037 |
17,825.3 |
92.6 |
2.89 |
9,062 |
Qwen3.5-27B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
96.5 |
287 |
2,970.2 |
16.1 |
263 |
16,286.6 |
36.1 |
2.82 |
14,342 |
Qwen3.5-27B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
736.8 |
2,227 |
3,022.2 |
114.6 |
2,040 |
17,793.6 |
140.6 |
2.83 |
14,387 |
Qwen3.5-2B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
1 |
41.6 |
287 |
6,886.3 |
12.0 |
262 |
21,929.8 |
123.6 |
2.39 |
1,623 |
Qwen3.5-2B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
8 |
412.4 |
2,227 |
5,399.3 |
82.0 |
2,036 |
24,813.9 |
384.0 |
2.40 |
1,629 |
Qwen3.5-2B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
11.5 |
287 |
24,978.3 |
11.8 |
262 |
22,222.2 |
246.6 |
2.44 |
1,559 |
Qwen3.5-2B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
76.8 |
2,227 |
28,982.7 |
82.6 |
2,040 |
24,691.4 |
718.4 |
2.37 |
1,564 |
Qwen3.5-4B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
1 |
97.3 |
287 |
2,945.2 |
12.1 |
263 |
21,786.5 |
79.9 |
2.53 |
1,914 |
Qwen3.5-4B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
8 |
970.9 |
2,227 |
2,293.5 |
82.9 |
2,037 |
24,570.0 |
257.7 |
2.52 |
1,890 |
Qwen3.5-4B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
23.4 |
287 |
12,229.4 |
11.7 |
262 |
22,471.9 |
135.7 |
2.54 |
2,797 |
Qwen3.5-4B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
163.0 |
2,227 |
13,665.8 |
82.9 |
2,037 |
24,570.0 |
408.9 |
2.54 |
2,816 |
Qwen3.5-9B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
1 |
171.0 |
287 |
1,676.1 |
16.1 |
262 |
16,286.6 |
57.5 |
2.78 |
2,945 |
Qwen3.5-9B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
8 |
1,560.3 |
2,227 |
1,427.2 |
113.7 |
2,037 |
17,921.1 |
202.6 |
2.80 |
2,952 |
Qwen3.5-9B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
1 |
32.7 |
287 |
8,760.5 |
16.1 |
262 |
16,286.6 |
93.0 |
2.71 |
4,783 |
Qwen3.5-9B |
VLM |
MTP |
NVFP4 / NVFP4 / FP16 |
8 |
235.4 |
2,227 |
9,458.1 |
113.6 |
2,039 |
17,953.3 |
329.5 |
2.72 |
4,842 |
Jetson AGX Orin 64GB#
Model |
Kind |
Mode |
Precision |
Batch |
Runtime Prefill (ms) |
Runtime Prefill Tok/Run |
Runtime Prefill (tok/s) |
ViT (ms) |
ViT Tok/Run |
ViT (tok/s) |
Generation (tok/s) |
Accept Rate |
GPU Mem (MB) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Qwen3-0.6B |
LLM |
Vanilla |
INT4 AWQ |
1 |
27.3 |
370 |
13,557.9 |
- |
- |
- |
177.2 |
- |
1,968 |
Qwen3-0.6B |
LLM |
Vanilla |
INT4 AWQ |
8 |
275.9 |
2,959 |
10,723.5 |
- |
- |
- |
584.0 |
- |
4,188 |
Qwen3-1.7B |
LLM |
Vanilla |
INT4 AWQ |
1 |
63.5 |
370 |
5,822.7 |
- |
- |
- |
95.5 |
- |
3,292 |
Qwen3-1.7B |
LLM |
Vanilla |
INT4 AWQ |
8 |
787.5 |
2,959 |
3,757.3 |
- |
- |
- |
406.7 |
- |
5,715 |
Qwen3-30B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
1 |
206.4 |
370 |
1,792.0 |
- |
- |
- |
55.2 |
- |
30,025 |
Qwen3-30B-A3B |
LLM |
Vanilla |
INT4 GPTQ |
8 |
2,094.1 |
2,959 |
1,412.9 |
- |
- |
- |
154.2 |
- |
31,613 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
INT4 AWQ |
1 |
150.6 |
364 |
2,416.5 |
- |
- |
- |
52.0 |
- |
4,933 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
INT4 AWQ |
8 |
1,952.8 |
2,911 |
1,490.6 |
- |
- |
- |
211.4 |
- |
8,194 |
Qwen3-8B |
LLM |
Vanilla |
INT4 AWQ |
1 |
273.9 |
370 |
1,350.5 |
- |
- |
- |
32.1 |
- |
8,237 |
Qwen3-8B |
LLM |
Vanilla |
INT4 AWQ |
8 |
3,561.2 |
2,959 |
830.8 |
- |
- |
- |
139.0 |
- |
11,229 |
Qwen3.5-0.8B-LLM |
LLM |
Vanilla |
INT4 AWQ |
1 |
124.2 |
377 |
3,033.3 |
- |
- |
- |
145.7 |
- |
2,329 |
Qwen3.5-0.8B-LLM |
LLM |
Vanilla |
INT4 AWQ |
8 |
939.4 |
3,013 |
3,207.4 |
- |
- |
- |
633.3 |
- |
3,333 |
Qwen3.5-27B-LLM |
LLM |
Vanilla |
INT4 AWQ |
1 |
1,335.8 |
377 |
282.0 |
- |
- |
- |
10.5 |
- |
24,136 |
Qwen3.5-27B-LLM |
LLM |
Vanilla |
INT4 AWQ |
8 |
16,963.3 |
3,013 |
177.6 |
- |
- |
- |
44.5 |
- |
26,293 |
Qwen3.5-2B-LLM |
LLM |
Vanilla |
INT4 AWQ |
1 |
161.9 |
377 |
2,326.5 |
- |
- |
- |
81.6 |
- |
4,197 |
Qwen3.5-2B-LLM |
LLM |
Vanilla |
INT4 AWQ |
8 |
1,447.5 |
3,013 |
2,081.7 |
- |
- |
- |
393.6 |
- |
5,117 |
Qwen3.5-35B-A3B-LLM |
LLM |
Vanilla |
INT4 GPTQ |
1 |
340.7 |
377 |
1,105.4 |
- |
- |
- |
31.0 |
- |
35,742 |
Qwen3.5-35B-A3B-LLM |
LLM |
Vanilla |
INT4 GPTQ |
8 |
2,807.2 |
3,013 |
1,073.4 |
- |
- |
- |
128.5 |
- |
36,654 |
Qwen3.5-4B-LLM |
LLM |
Vanilla |
INT4 AWQ |
1 |
307.0 |
377 |
1,226.9 |
- |
- |
- |
45.4 |
- |
6,147 |
Qwen3.5-4B-LLM |
LLM |
Vanilla |
INT4 AWQ |
8 |
3,428.1 |
3,013 |
879.0 |
- |
- |
- |
187.7 |
- |
7,659 |
Qwen3.5-9B-LLM |
LLM |
Vanilla |
INT4 AWQ |
1 |
437.5 |
377 |
860.9 |
- |
- |
- |
27.9 |
- |
9,983 |
Qwen3.5-9B-LLM |
LLM |
Vanilla |
INT4 AWQ |
8 |
5,122.9 |
3,013 |
588.2 |
- |
- |
- |
125.8 |
- |
11,278 |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
48.0 |
283 |
5,883.5 |
38.7 |
262 |
6,775.1 |
95.7 |
- |
8,235 |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
408.3 |
2,196 |
5,378.2 |
276.1 |
2,039 |
7,385.5 |
388.8 |
- |
10,304 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
118.6 |
283 |
2,384.0 |
39.0 |
262 |
6,724.9 |
52.2 |
- |
9,863 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
1,024.0 |
2,196 |
2,144.3 |
277.4 |
2,038 |
7,347.5 |
176.4 |
- |
12,511 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
213.2 |
283 |
1,325.8 |
56.5 |
262 |
4,646.8 |
32.2 |
- |
13,133 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
1,842.2 |
2,196 |
1,191.9 |
398.8 |
2,039 |
5,112.5 |
141.0 |
- |
16,014 |
Qwen3.5-0.8B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
94.0 |
287 |
3,050.3 |
13.0 |
262 |
20,161.3 |
141.4 |
- |
4,549 |
Qwen3.5-0.8B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
501.1 |
2,227 |
4,444.2 |
89.9 |
2,038 |
22,675.7 |
451.8 |
- |
5,058 |
Qwen3.5-27B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
1,037.4 |
287 |
276.3 |
53.5 |
262 |
4,906.8 |
10.5 |
- |
24,048 |
Qwen3.5-27B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
8,777.6 |
2,227 |
253.7 |
382.1 |
2,038 |
5,333.3 |
44.2 |
- |
26,372 |
Qwen3.5-2B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
120.8 |
287 |
2,372.0 |
36.6 |
262 |
7,168.5 |
81.0 |
- |
6,886 |
Qwen3.5-2B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
766.8 |
2,227 |
2,903.9 |
264.7 |
2,038 |
7,698.2 |
303.9 |
- |
7,456 |
Qwen3.5-35B-A3B |
VLM |
Vanilla |
INT4 GPTQ / FP16 |
1 |
297.1 |
287 |
964.7 |
52.8 |
262 |
4,970.2 |
30.8 |
- |
35,810 |
Qwen3.5-35B-A3B |
VLM |
Vanilla |
INT4 GPTQ / FP16 |
8 |
1,537.3 |
2,227 |
1,448.6 |
379.6 |
2,038 |
5,367.7 |
131.4 |
- |
36,733 |
Qwen3.5-4B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
234.9 |
287 |
1,220.0 |
36.6 |
262 |
7,158.2 |
45.3 |
- |
8,638 |
Qwen3.5-4B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
1,815.4 |
2,227 |
1,226.7 |
265.5 |
2,038 |
7,674.6 |
174.8 |
- |
9,798 |
Qwen3.5-9B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
341.5 |
287 |
839.2 |
53.2 |
262 |
4,931.0 |
27.9 |
- |
12,247 |
Qwen3.5-9B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
8 |
2,672.0 |
2,227 |
833.4 |
380.9 |
2,038 |
5,350.5 |
121.7 |
- |
13,410 |
Qwen3-1.7B |
LLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ |
1 |
63.8 |
370 |
5,797.2 |
- |
- |
- |
152.4 |
3.87 |
3,362 |
Qwen3-1.7B |
LLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ |
8 |
784.4 |
2,959 |
3,772.0 |
- |
- |
- |
250.7 |
3.85 |
6,949 |
Qwen3-8B |
LLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ |
1 |
274.3 |
370 |
1,348.6 |
- |
- |
- |
57.8 |
4.18 |
8,349 |
Qwen3-8B |
LLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ |
8 |
3,567.7 |
2,959 |
829.3 |
- |
- |
- |
70.4 |
4.16 |
12,969 |
Qwen2.5-VL-7B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
1 |
255.2 |
376 |
1,472.3 |
86.1 |
344 |
3,998.4 |
67.2 |
5.05 |
12,218 |
Qwen2.5-VL-7B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
8 |
2,279.8 |
2,919 |
1,280.2 |
671.2 |
2,675 |
3,985.7 |
82.5 |
4.97 |
14,145 |
Qwen3-VL-4B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
1 |
118.0 |
283 |
2,396.0 |
39.1 |
262 |
6,711.4 |
98.3 |
5.05 |
10,310 |
Qwen3-VL-4B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
8 |
1,025.0 |
2,196 |
2,142.2 |
276.8 |
2,038 |
7,363.8 |
147.9 |
5.07 |
13,669 |
Qwen3-VL-8B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
1 |
213.6 |
283 |
1,323.0 |
56.6 |
262 |
4,633.9 |
39.1 |
2.81 |
13,932 |
Qwen3-VL-8B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
8 |
1,843.2 |
2,196 |
1,191.3 |
399.6 |
2,038 |
5,099.4 |
48.1 |
2.83 |
17,413 |
Qwen3.5-0.8B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
1 |
92.7 |
287 |
3,092.1 |
13.0 |
262 |
20,242.9 |
138.9 |
2.19 |
5,183 |
Qwen3.5-0.8B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
8 |
493.8 |
2,227 |
4,509.3 |
90.1 |
2,038 |
22,624.4 |
413.7 |
2.20 |
6,396 |
Qwen3.5-27B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
1 |
1,038.2 |
287 |
276.1 |
53.2 |
262 |
4,931.0 |
18.3 |
2.89 |
25,664 |
Qwen3.5-27B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
8 |
8,790.5 |
2,227 |
253.3 |
383.0 |
2,038 |
5,322.0 |
69.5 |
2.87 |
33,332 |
Qwen3.5-2B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
1 |
120.4 |
287 |
2,381.6 |
36.6 |
262 |
7,173.6 |
86.4 |
2.41 |
8,027 |
Qwen3.5-2B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
8 |
762.3 |
2,227 |
2,921.4 |
265.6 |
2,038 |
7,674.6 |
293.7 |
2.39 |
9,382 |
Qwen3.5-4B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
1 |
234.7 |
287 |
1,221.4 |
36.7 |
262 |
7,142.9 |
54.6 |
2.53 |
10,181 |
Qwen3.5-4B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
8 |
1,811.5 |
2,227 |
1,229.3 |
265.9 |
2,037 |
7,662.8 |
204.4 |
2.50 |
12,976 |
Qwen3.5-9B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
1 |
341.6 |
287 |
839.1 |
53.0 |
262 |
4,952.9 |
40.4 |
2.82 |
14,598 |
Qwen3.5-9B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
8 |
2,670.7 |
2,227 |
833.8 |
379.2 |
2,038 |
5,373.5 |
159.9 |
2.78 |
17,548 |
Jetson Orin NX 16GB#
Model |
Kind |
Mode |
Precision |
Batch |
Runtime Prefill (ms) |
Runtime Prefill Tok/Run |
Runtime Prefill (tok/s) |
ViT (ms) |
ViT Tok/Run |
ViT (tok/s) |
Generation (tok/s) |
Accept Rate |
GPU Mem (MB) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Qwen3-0.6B |
LLM |
Vanilla |
INT4 AWQ |
1 |
52.4 |
370 |
7,064.6 |
- |
- |
- |
110.8 |
- |
2,027 |
Qwen3-1.7B |
LLM |
Vanilla |
INT4 AWQ |
1 |
127.4 |
370 |
2,903.3 |
- |
- |
- |
58.8 |
- |
3,282 |
Qwen3-4B-Instruct-2507 |
LLM |
Vanilla |
INT4 AWQ |
1 |
306.8 |
364 |
1,186.0 |
- |
- |
- |
30.4 |
- |
4,914 |
Qwen3-8B |
LLM |
Vanilla |
INT4 AWQ |
1 |
615.1 |
370 |
601.3 |
- |
- |
- |
18.6 |
- |
8,210 |
Qwen3.5-0.8B-LLM |
LLM |
Vanilla |
INT4 AWQ |
1 |
174.8 |
377 |
2,154.5 |
- |
- |
- |
88.2 |
- |
2,286 |
Qwen3.5-2B-LLM |
LLM |
Vanilla |
INT4 AWQ |
1 |
249.4 |
377 |
1,510.3 |
- |
- |
- |
48.3 |
- |
4,235 |
Qwen3.5-4B-LLM |
LLM |
Vanilla |
INT4 AWQ |
1 |
535.8 |
377 |
703.0 |
- |
- |
- |
26.2 |
- |
6,093 |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
98.6 |
283 |
2,866.0 |
81.6 |
262 |
3,214.4 |
58.9 |
- |
4,406 |
Qwen3-VL-4B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
238.5 |
283 |
1,184.8 |
83.3 |
262 |
3,148.6 |
30.8 |
- |
5,990 |
Qwen3-VL-8B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
455.2 |
283 |
620.9 |
119.6 |
262 |
2,193.0 |
18.8 |
- |
9,092 |
Qwen3.5-0.8B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
136.6 |
287 |
2,099.0 |
26.7 |
262 |
9,832.8 |
87.6 |
- |
2,704 |
Qwen3.5-2B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
193.6 |
287 |
1,480.4 |
77.5 |
262 |
3,386.4 |
48.0 |
- |
4,704 |
Qwen3.5-4B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
414.6 |
287 |
691.3 |
79.0 |
262 |
3,321.2 |
25.9 |
- |
6,372 |
Qwen3-1.7B |
LLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ |
1 |
129.0 |
370 |
2,867.6 |
- |
- |
- |
86.5 |
3.87 |
3,394 |
Qwen3-8B |
LLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ |
1 |
651.4 |
370 |
567.8 |
- |
- |
- |
29.7 |
4.14 |
8,358 |
Qwen2.5-VL-7B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
1 |
505.7 |
376 |
742.9 |
200.7 |
344 |
1,715.6 |
34.7 |
5.05 |
9,208 |
Qwen3-VL-4B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
1 |
237.8 |
283 |
1,188.7 |
83.1 |
262 |
3,156.6 |
56.2 |
5.11 |
6,427 |
Qwen3-VL-8B-Instruct |
VLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ / FP16 |
1 |
488.9 |
283 |
578.0 |
121.0 |
262 |
2,167.8 |
20.4 |
2.84 |
9,565 |
Qwen3.5-0.8B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
1 |
135.8 |
287 |
2,111.0 |
26.7 |
262 |
9,813.5 |
77.7 |
2.16 |
3,296 |
Qwen3.5-2B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
1 |
193.3 |
287 |
1,482.8 |
77.8 |
262 |
3,372.7 |
46.8 |
2.38 |
5,758 |
Qwen3.5-4B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
1 |
410.7 |
287 |
697.8 |
78.0 |
262 |
3,363.6 |
28.9 |
2.52 |
7,838 |
Jetson Orin Nano 8GB#
Model |
Kind |
Mode |
Precision |
Batch |
Runtime Prefill (ms) |
Runtime Prefill Tok/Run |
Runtime Prefill (tok/s) |
ViT (ms) |
ViT Tok/Run |
ViT (tok/s) |
Generation (tok/s) |
Accept Rate |
GPU Mem (MB) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Qwen3-0.6B |
LLM |
Vanilla |
INT4 AWQ |
1 |
94.7 |
370 |
3,904.2 |
- |
- |
- |
69.4 |
- |
1,999 |
Qwen3-1.7B |
LLM |
Vanilla |
INT4 AWQ |
1 |
231.5 |
370 |
1,597.9 |
- |
- |
- |
36.3 |
- |
3,286 |
Qwen3.5-0.8B-LLM |
LLM |
Vanilla |
INT4 AWQ |
1 |
316.3 |
377 |
1,190.9 |
- |
- |
- |
55.0 |
- |
2,294 |
Qwen3.5-2B-LLM |
LLM |
Vanilla |
INT4 AWQ |
1 |
445.0 |
377 |
846.5 |
- |
- |
- |
29.8 |
- |
4,145 |
Qwen3-VL-2B-Instruct |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
181.7 |
283 |
1,555.1 |
150.6 |
262 |
1,741.6 |
36.3 |
- |
4,444 |
Qwen3.5-0.8B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
244.0 |
287 |
1,174.8 |
48.8 |
262 |
5,376.3 |
54.4 |
- |
2,656 |
Qwen3.5-2B |
VLM |
Vanilla |
INT4 AWQ / FP16 |
1 |
349.8 |
287 |
819.3 |
142.4 |
262 |
1,842.6 |
29.4 |
- |
4,649 |
Qwen3-1.7B |
LLM |
EAGLE3 |
INT4 AWQ / INT4 AWQ |
1 |
232.2 |
370 |
1,593.0 |
- |
- |
- |
48.7 |
3.87 |
3,398 |
Qwen3.5-0.8B |
VLM |
MTP |
INT4 AWQ / INT4 AWQ / FP16 |
1 |
244.1 |
287 |
1,174.1 |
48.7 |
262 |
5,387.9 |
46.9 |
2.16 |
3,278 |
v0.7.1 Results#
SDK Version: TensorRT Edge-LLM 0.7.1 | TensorRT: 10.13.3.9
LLM — Vanilla Decoding#
Model |
Precision |
Batch |
Prefill (ms) |
Prefill Tokens |
Prefill (tok/s) |
Generation (tok/s) |
GPU Mem (MB) |
|---|---|---|---|---|---|---|---|
Qwen3-1.7B |
NVFP4 |
1 |
11.2 |
370 |
32,990 |
173.2 |
1,531 |
Qwen3-1.7B |
NVFP4 |
8 |
132.5 |
2,959 |
22,324 |
935.8 |
1,475 |
Qwen3-30B-A3B-GPTQ-Int4 |
INT4 GPTQ |
1 |
133.2 |
370 |
2,777 |
72.6 |
15,916 |
Qwen3-30B-A3B-GPTQ-Int4 |
INT4 GPTQ |
8 |
1,344.5 |
2,959 |
2,201 |
215.9 |
15,894 |
Nemotron-3-Nano-4B |
NVFP4 |
1 |
127.4 |
383 |
3,004 |
64.7 |
3,568 |
Nemotron-3-Nano-4B |
NVFP4 |
8 |
986.6 |
3,062 |
3,104 |
312.9 |
3,592 |
Vision Language Model — Vanilla Decoding#
Model |
LLM Prec |
ViT Prec |
Batch |
Prefill (ms) |
Prefill Tokens |
Prefill (tok/s) |
ViT Time (ms) |
ViT Tok/Run |
ViT (tok/s) |
Generation (tok/s) |
GPU Mem (MB) |
|---|---|---|---|---|---|---|---|---|---|---|---|
Qwen2.5-VL-7B-Instruct |
NVFP4 |
FP16 |
1 |
31.0 |
376 |
12,124 |
27.8 |
344 |
12,392 |
49.5 |
5,224 |
Qwen3.5-0.8B |
NVFP4 |
FP16 |
1 |
9.4 |
287 |
30,410 |
4.3 |
262 |
60,606 |
287.0 |
1,192 |
Qwen3.5-2B |
NVFP4 |
FP16 |
1 |
12.7 |
287 |
22,550 |
10.5 |
262 |
25,000 |
164.4 |
1,694 |
Qwen3.5-27B |
NVFP4 |
FP16 |
1 |
103.3 |
287 |
2,775 |
15.0 |
262 |
17,483 |
16.1 |
14,725 |
Nemotron-3-Nano-Omni-30B-A3B |
NVFP4 |
FP16 |
1 |
226.0 |
1,663 |
7,358 |
121.3 |
1,635 |
13,477 |
31.3 |
20,327 |
LLM — EAGLE3 Speculative Decoding#
Draft Models#
Base Model |
Draft Model |
Source |
|---|---|---|
Qwen3-1.7B |
Qwen3-1.7B_eagle3 |
Note: Both base and draft models are quantized to NVFP4.
Model |
Base Prec |
Draft Prec |
Batch |
Prefill (ms) |
Prefill Tokens |
Generation (tok/s) |
Accept Rate |
Speedup |
GPU Mem (MB) |
|---|---|---|---|---|---|---|---|---|---|
Qwen3-1.7B |
NVFP4 |
NVFP4 |
1 |
12.2 |
370 |
339.0 |
3.7 |
1.96x |
1,534 |
Qwen3-1.7B |
NVFP4 |
NVFP4 |
8 |
132.3 |
2,959 |
984.9 |
3.7 |
1.05x |
1,466 |
Vision Language Model — MTP Speculative Decoding#
Note: MTP uses the model’s built-in draft heads; no external draft checkpoint is required. Highlight: MTP is the main v0.7.1 performance improvement, increasing Qwen3.5 VLM BS=1 generation throughput by 1.21x to 2.12x over vanilla decoding.
Model |
Base Prec |
Draft Prec |
ViT Prec |
Batch |
Prefill (ms) |
Prefill Tokens |
ViT Time (ms) |
ViT Tok/Run |
ViT (tok/s) |
Generation (tok/s) |
Accept Rate |
GPU Mem (MB) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
Qwen3.5-0.8B |
NVFP4 |
NVFP4 |
FP16 |
1 |
9.2 |
287 |
4.2 |
263 |
62,500 |
348.5 |
2.1 |
1,210 |
Qwen3.5-0.8B |
NVFP4 |
NVFP4 |
FP16 |
8 |
69.0 |
2,227 |
27.6 |
2,042 |
74,074 |
1,056.7 |
2.2 |
1,375 |
Qwen3.5-2B |
NVFP4 |
NVFP4 |
FP16 |
1 |
13.8 |
287 |
10.9 |
262 |
24,096 |
236.9 |
2.4 |
1,662 |
Qwen3.5-2B |
NVFP4 |
NVFP4 |
FP16 |
8 |
89.7 |
2,227 |
75.1 |
2,040 |
27,174 |
787.2 |
2.4 |
1,647 |
Qwen3.5-27B |
NVFP4 |
NVFP4 |
FP16 |
1 |
111.1 |
287 |
14.4 |
262 |
18,215 |
34.2 |
2.8 |
14,680 |
Qwen3.5-27B |
NVFP4 |
NVFP4 |
FP16 |
8 |
811.0 |
2,227 |
108.2 |
2,038 |
18,832 |
146.7 |
2.8 |
14,705 |
v0.7.0 Results#
SDK Version: TensorRT Edge-LLM 0.7.0 | TensorRT: 10.13
LLM — Vanilla Decoding#
Model |
Precision |
Batch |
Prefill (ms) |
Prefill Tokens |
Prefill (tok/s) |
Generation (tok/s) |
GPU Mem (MB) |
|---|---|---|---|---|---|---|---|
Qwen3-1.7B |
NVFP4 |
1 |
13.9 |
370 |
26,683 |
170.4 |
1,453 |
Qwen3-1.7B |
NVFP4 |
8 |
150.5 |
2,959 |
19,663 |
798.8 |
1,491 |
Qwen3-30B-A3B-GPTQ-Int4 |
INT4 GPTQ |
1 |
125.3 |
370 |
2,951 |
81.3 |
15,938 |
Qwen3-30B-A3B-GPTQ-Int4 |
INT4 GPTQ |
8 |
1,342.2 |
2,959 |
2,204 |
223.2 |
15,961 |
Nemotron-3-Nano-4B |
NVFP4 |
1 |
126.8 |
383 |
3,018 |
65.4 |
3,647 |
Nemotron-3-Nano-4B |
NVFP4 |
8 |
1,017.6 |
3,062 |
3,009 |
315.4 |
3,684 |
Vision Language Model — Vanilla Decoding#
Model |
LLM Prec |
ViT Prec |
Prefill (ms) |
Prefill Tokens |
Prefill (tok/s) |
Generation (tok/s) |
GPU Mem (MB) |
|---|---|---|---|---|---|---|---|
Qwen3.5-0.8B |
NVFP4 |
FP16 |
7.0 |
753 |
107,571 |
232.2 |
1,052 |
Qwen3.5-2B |
NVFP4 |
FP16 |
13.8 |
753 |
54,565 |
111.0 |
1,671 |
Qwen3.5-27B |
NVFP4 |
FP16 |
122.6 |
753 |
6,143 |
10.5 |
14,985 |
Nemotron-3-Nano-Omni-30B-A3B |
NVFP4 |
FP16 |
846.7 |
1,663 |
1,964 |
24.5 |
20,267 |
LLM — EAGLE3 Speculative Decoding#
Draft Models#
Base Model |
Draft Model |
Source |
|---|---|---|
Qwen3-1.7B |
Qwen3-1.7B_eagle3 |
Note: Both base and draft models are quantized to NVFP4.
Model |
Base Prec |
Draft Prec |
Batch |
Prefill (ms) |
Prefill Tokens |
Generation (tok/s) |
Accept Rate |
Speedup |
|---|---|---|---|---|---|---|---|---|
Qwen3-1.7B |
NVFP4 |
NVFP4 |
1 |
14.5 |
370 |
312.4 |
3.75 |
1.83x |
Qwen3-1.7B |
NVFP4 |
NVFP4 |
8 |
153.5 |
2,959 |
828.8 |
3.73 |
1.04x |
v0.4.0 Results#
SDK Version: TensorRT Edge-LLM 0.4.0 | TensorRT: 10.13
LLM — Vanilla Decoding#
Model |
Precision |
Batch |
Prefill (ms) |
Prefill Tokens |
Prefill (tok/s) |
Generation (tok/s) |
|---|---|---|---|---|---|---|
Llama-3.1-8B-Instruct |
INT4 AWQ |
1 |
215.5 |
383 |
1,777 |
50.8 |
Llama-3.1-8B-Instruct |
INT4 AWQ |
8 |
2737.4 |
3064 |
1,119 |
135.3 |
Llama-3.1-8B-Instruct |
NVFP4 |
1 |
31.0 |
383 |
12,355 |
54.9 |
Llama-3.1-8B-Instruct |
NVFP4 |
8 |
387.6 |
3064 |
7,905 |
308.7 |
Qwen3-0.6B |
INT4 AWQ |
1 |
21.0 |
366 |
17,429 |
270.2 |
Qwen3-0.6B |
INT4 AWQ |
8 |
241.8 |
2927 |
12,104 |
828.0 |
Qwen3-0.6B |
NVFP4 |
1 |
8.8 |
366 |
41,591 |
318.6 |
Qwen3-0.6B |
NVFP4 |
8 |
95.4 |
2927 |
30,681 |
1562.4 |
Qwen3-4B-Instruct-2507 |
INT4 AWQ |
1 |
116.2 |
364 |
3,133 |
76.4 |
Qwen3-4B-Instruct-2507 |
INT4 AWQ |
8 |
1502.3 |
2911 |
1,938 |
240.3 |
Qwen3-4B-Instruct-2507 |
NVFP4 |
1 |
22.9 |
364 |
15,895 |
90.2 |
Qwen3-4B-Instruct-2507 |
NVFP4 |
8 |
301.9 |
2911 |
9,642 |
507.4 |
Qwen3-8B |
INT4 AWQ |
1 |
212.0 |
366 |
1,726 |
47.7 |
Qwen3-8B |
INT4 AWQ |
8 |
2719.1 |
2927 |
1,076 |
162.3 |
Qwen3-8B |
NVFP4 |
1 |
32.8 |
366 |
11,159 |
53.7 |
Qwen3-8B |
NVFP4 |
8 |
425.8 |
2927 |
6,874 |
372.2 |
Vision Language Model — Vanilla Decoding#
Model |
LLM Prec |
ViT Prec |
Prefill (ms) |
Prefill Tokens |
Prefill (tok/s) |
ViT Time (ms) |
ViT Tok/Run |
ViT (tok/s) |
Generation (tok/s) |
|---|---|---|---|---|---|---|---|---|---|
Qwen2.5-VL-7B-Instruct |
INT4 AWQ |
FP16 |
195.1 |
376 |
1,927 |
51.1 |
344 |
6,732 |
53.1 |
Qwen2.5-VL-7B-Instruct |
INT4 AWQ |
FP8 |
195.1 |
376 |
1,927 |
42.7 |
344 |
8,056 |
53.1 |
Qwen2.5-VL-7B-Instruct |
NVFP4 |
FP16 |
25.7 |
376 |
14,631 |
51.0 |
344 |
6,745 |
57.7 |
Qwen2.5-VL-7B-Instruct |
NVFP4 |
FP8 |
25.7 |
376 |
14,631 |
42.6 |
344 |
8,075 |
57.6 |
Qwen3-VL-2B-Instruct |
INT4 AWQ |
FP16 |
39.4 |
283 |
7,183 |
19.0 |
262 |
13,789 |
144.4 |
Qwen3-VL-2B-Instruct |
INT4 AWQ |
FP8 |
39.4 |
283 |
7,183 |
15.4 |
262 |
17,013 |
144.7 |
Qwen3-VL-2B-Instruct |
NVFP4 |
FP16 |
10.1 |
283 |
28,020 |
19.0 |
262 |
13,789 |
180.8 |
Qwen3-VL-2B-Instruct |
NVFP4 |
FP8 |
10.1 |
283 |
28,020 |
15.5 |
262 |
16,903 |
181.0 |
Note: ViT time = per-token ViT latency x image tokens per run. FP8 ViT reduces visual encoder time by ~17% compared to FP16 with negligible impact on generation throughput.
LLM — EAGLE3 Speculative Decoding#
Draft Models#
Base Model |
Draft Model |
Source |
|---|---|---|
Llama-3.1-8B-Instruct |
EAGLE3-LLaMA3.1-Instruct-8B |
|
Qwen3-8B |
qwen3_8b_eagle3 |
Note: Both base and draft models are quantized to the same precision (INT4 AWQ or NVFP4) as listed in the table below.
Model |
Base Prec |
Draft Prec |
Batch |
Prefill (ms) |
Prefill Tokens |
Generation (tok/s) |
Accept Rate |
Speedup |
|---|---|---|---|---|---|---|---|---|
Llama-3.1-8B-Instruct |
INT4 AWQ |
INT4 AWQ |
1 |
215.2 |
382 |
81.0 |
5.25 |
1.59x |
Llama-3.1-8B-Instruct |
INT4 AWQ |
INT4 AWQ |
8 |
2735.5 |
3056 |
118.0 |
5.21 |
0.87x |
Llama-3.1-8B-Instruct |
NVFP4 |
NVFP4 |
1 |
30.8 |
382 |
189.2 |
5.21 |
3.45x |
Llama-3.1-8B-Instruct |
NVFP4 |
NVFP4 |
8 |
413.1 |
3056 |
484.7 |
5.15 |
1.57x |
Qwen3-8B |
INT4 AWQ |
INT4 AWQ |
1 |
212.2 |
366 |
66.1 |
4.36 |
1.39x |
Qwen3-8B |
INT4 AWQ |
INT4 AWQ |
8 |
2719.1 |
2927 |
99.1 |
4.31 |
0.61x |
Qwen3-8B |
NVFP4 |
NVFP4 |
1 |
33.1 |
366 |
151.7 |
4.26 |
2.82x |
Qwen3-8B |
NVFP4 |
NVFP4 |
8 |
429.1 |
2927 |
457.7 |
4.25 |
1.23x |
Note: EAGLE3 speculative decoding provides the greatest speedup at BS=1 (latency-bound). At BS=8, base model compute is already well-utilized, limiting speculative acceleration. See Speculative Decoding for setup instructions.
Vision Language Model — EAGLE3 Speculative Decoding#
Draft Models#
Base Model |
Draft Model |
Source |
|---|---|---|
Qwen2.5-VL-7B-Instruct |
qwen2.5-vl-7b-eagle3-sgl |
Note: Both base and draft models are quantized to the same precision as listed in the table below.
Model |
Base Prec |
Draft Prec |
ViT Prec |
Prefill (ms) |
Prefill Tokens |
Generation (tok/s) |
Accept Rate |
Speedup |
|---|---|---|---|---|---|---|---|---|
Qwen2.5-VL-7B-Instruct |
INT4 AWQ |
INT4 AWQ |
FP16 |
195.1 |
376 |
57.3 |
3.66 |
1.08x |
Qwen2.5-VL-7B-Instruct |
NVFP4 |
NVFP4 |
FP16 |
25.8 |
376 |
149.6 |
3.82 |
2.59x |
Qwen2.5-VL-7B-Instruct |
NVFP4 |
NVFP4 |
FP8 |
32.8 |
376 |
117.3 |
3.76 |
2.04x |
Key Observations#
v0.9.0#
Current release baseline: v0.9.0 is the current performance baseline and supersedes the v0.8.0 regression note. Runtime tables include batch size, dataset, prefill sequence length/time, decode throughput, speculative acceptance rate, and peak GPU memory.
Speculative decoding coverage: EAGLE3, MTP, and DFlash rows are included for supported v0.9.0 engines. Orin platforms use the externalized INT4 subset supported by each memory target.
v0.8.0#
All release devices are covered: v0.8.0 adds Jetson AGX Thor, Jetson AGX Orin 64GB, Jetson Orin NX 16GB, and Jetson Orin Nano 8GB results from the release benchmark outputs, benchmarked under JetPack 7.2.
First
llm_benchprefill metrics: AGX Thor includes parsedllm_benchprefill measurements atinputLen=2048. Qwen3-0.6B NVFP4 reaches 91,469.3 tok/s at BS=1; Qwen3-1.7B EAGLE3 NVFP4 reaches 66,929.4 tok/s at BS=1.Speculative decode remains platform-dependent: EAGLE3 and MTP report strong acceptance rates, but generation throughput depends heavily on model size, precision, and platform memory bandwidth.
v0.7.1#
MTP speculative decoding: This is the v0.7.1 performance highlight. Qwen3.5 MTP improves BS=1 generation throughput by 1.21x for 0.8B, 1.44x for 2B, and 2.12x for 27B over vanilla decoding, with BS=8 throughput up to 1,056.7 tok/s for Qwen3.5-0.8B.
v0.7.0#
MoE support: Qwen3-30B-A3B-GPTQ-Int4 (MoE, 3B active params out of 30B) achieves 81.3 tok/s at BS=1 and 223.2 tok/s at BS=8 with INT4 GPTQ, demonstrating efficient sparse model inference on edge.
Small model throughput: Qwen3-1.7B with NVFP4 delivers 170.4 tok/s at BS=1 and 798.8 tok/s at BS=8, suitable for latency-sensitive edge applications.
Qwen3.5 VLM family: Ranges from 232.2 tok/s (0.8B) to 10.5 tok/s (27B), providing a scalable VLM option across memory and throughput budgets.
Nemotron-3-Nano-Omni-30B-A3B: The first audio+video multimodal model benchmarked, achieving 24.5 tok/s generation at 20 GB GPU memory.
v0.4.0#
NVFP4 delivers highest throughput: NVFP4 achieves 1.1–2.3x higher generation throughput than INT4 AWQ, with substantially faster prefill (e.g., 31 ms vs 216 ms for Llama-3.1-8B at BS=1).
EAGLE3 at BS=1 provides meaningful speedup: 1.4–3.5x for LLMs, best for Llama-3.1-8B NVFP4 (3.45x). The draft model acceptance rate is high for Llama (~5.2 tokens/step) and moderate for Qwen3-8B (~4.3 tokens/step).
EAGLE3 at BS=8 has limited benefit: At high batch sizes, base model compute is already well-utilized. Speedup drops to <1x for INT4 AWQ and 1.2–1.6x for NVFP4.
Qwen3-0.6B achieves the highest throughput: 1562 tok/s at BS=8 with NVFP4 — a lightweight model well-suited for latency-sensitive edge applications.
General#
Benchmarks use default TensorRT Edge-LLM inference settings on the listed device. Production performance may vary with system-level tuning (power mode, memory configuration, thermal management).