Performance Benchmarking#

The following results compare fine-tuning and inference performance across different NVIDIA GPU configurations.

Technical terms used on this page are defined first. For the performance results, start with the best observed configuration for your hardware, then use the metric tabs and filters to compare throughput, relative cost, and step time.

Nomenclature#

GBS: Global Batch Size
MBS: Micro Batch Size
TP: Tensor Parallel Size
PP: Pipeline Parallel Size
CP: Context Parallel Size
EP: Expert Parallel Size
ETP: Expert Tensor Parallel Size
DDP: Distributed Data Parallel
FSDP: Fully Sharded Data Parallel (FSDP2 in AutoModel and Megatron FSDP for mbridge)
AC: Activation Checkpointing (a memory-saving technique)
Packed length: Maximum tokens per packed fine-tuning sample

Fine-Tuning Performance#

Fine-tuning performance is reported using the following metrics:

Time per global step (s): Optimizer step mean duration to process one global batch size.
Tokens/s/GPU: Token throughput divided by the number of GPUs.

Best observed configurations#

All fine-tuning configurations were optimized for a single eight-GPU node. This does not mean we recommend only a single node for training.

The chart compares the metrics corresponding to the best possible configuration for different GPUs, while the table tab shows the details for those configuration. The download tab includes performance metric details for all tested configurations.

Step Time is the mean duration for an optimizer step processing one global batch size of 128.

Relative cost per 1M tokens combines measured full-node throughput with an hourly eight-GPU price index. A100 is used as our reference baseline at $1/hour; H100 and B200 are 1.7x and 3.5x the A100 price, respectively.

NOTE: Results are intended for relative comparison only. Configurations are absent when a better-performing recipe was available or when the model did not fit into GPU under that configuration.

Example training time#

The metrics above are per-step measurements. The following example converts them into wall-clock time, to show how long a complete fine-tuning run takes on a given hardware setup. Each run is split into jobs of 200 steps, so the estimated total time is the number of jobs needed to reach the expected step count multiplied by the average time per job.

Table 1 Training Period Example#
Training Stack	Compute	Steps Per Job	Average Time Per Job	Expected Training Steps	Estimated Total Time
AutoModel	8 nodes / 64 A100 80GB GPUs	200	3 hours	3,000 – 5,000	2 – 3 days
Megatron-Bridge	8 nodes / 64 A100 80GB GPUs	200	2 hours	3,000 – 5,000	1 – 2 days

These estimates correspond to the tennis dataset below. Training time scales with the number of steps, so a smaller dataset or fewer epochs shortens the run proportionally.

Table 2 Tennis Dataset Size#
Data Split	MCQ Pairs	QA Pairs	Total Pairs	Unique Video Clips
Training	1,226,081	86,048	1,312,129	43,084
Validation	64,401	4,530	68,931	2,267

Inference Performance#

After fine-tuning, we run inference with the resulting model. Inference performance is summarized using the following metrics:

Requests/s: Number of completed real-video requests divided by the measured wall-clock time.
Mean E2E duration (s/request): Average time per request from the start of media preprocessing through the completion of generation. Model loading and warm-up are excluded.

The benchmark uses a fixed set of 64 requests and the reported results are averaged. Each request consists of one video clip (with audio), a text question/prompt, and the generated response, with average input text prompt length of 884 tokens (~9k tokens for all of the visual, audio, and text inputs) and output length of 24 tokens.

All reported inference results use a single GPU with TP=1, EP=1, and ETP=1. These topology values are fixed for the table below.

System	# GPUs	Recommended Stack	Requests/s	Mean E2E duration (s/request)
DGX Spark	1	Megatron-Bridge	0.275	3.63
RTX PRO 6000	1	Megatron-Bridge	0.323	3.09