Basketball#

Basketball is a reference sport for these playbooks. This page is a reproduction reference for Nemotron-3-Nano-Omni fine-tuned on BARD: it answers the question types below on short basketball broadcast clips, with Hugging Face weights on Published Checkpoints. The walkthrough covers dataset, data preparation, training, and evaluation, with basketball-specific settings on top of the generic configs in the code repo.

Dataset#

BARD is a public basketball dataset of 14,676 labelled broadcast basketball clips from 60 games. Clips are cut on play-by-play boundaries. Each clip carries an ordered list of action labels, with the acting player’s jersey number and colour, the action, its result, and the assister, plus a caption that states every action in the clip, one sentence per action. The complete dataset (about 324 GB) is required for data preparation.

Data Preparation#

The example follows the generic pipeline in Data Curation. The data generator applies the applicable templates (from 14 MCQ and 2 open-ended templates) to every clip and results in 175,193 data points from 14,676 clips.

Splits are assigned such that every row from the same clip stays together. We get the following splits:

train / validation — clips from games used for training; validation is a held-out slice of rows from those clips
test-seen — held-out clips from games that also appear in train (new clip, same games)
test-unseen — nine whole games never used in train, validation, or test-seen

cd avlm/data_prep_example/basketball
python prepare_training_data_pipeline.py
# output: training_basketball_data_output/bard_cat_<split>.jsonl

README (start here): avlm/data_prep_example/basketball/README.md
Question taxonomy: CATEGORICAL_EVAL.MD, the 16 templates with their super and fine categories, per-class ground truth, and data-quality gates
Per-class examples and baselines: TEMPLATE_EXAMPLES_CATEGORICAL.MD

Training#

Training is full SFT with NeMo AutoModel. Start from the generic multinode SFT config and change the settings below.

Setting	Value
Base checkpoint	nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 Perform a one-time preprocessing step by running avlm/utils/fold_video_temporal.py on the base checkpoint to disable temporal merge in the vision encoder so that each frame is encoded on its own. Set `pretrained_model_name_or_path` to point to the prepared base checkpoint and set `video_temporal_patch_dim: 1` and `video_target_num_patches: 4096`
Vision encoder	`freeze_vision_tower: false` and `freeze_embeddings: true`. The audio encoder stays frozen and the language model stays trainable similar to the generic config
Frame sampling	2 fps, up to 32 frames. Per-frame encoding doubles the tokens per frame.
Sequence packing	pack size 40,960, packing ratio 0.85
Data	`bard_cat_train.jsonl` and `bard_cat_val.jsonl` from the data preparation step

Everything else stays as in the generic config. The released model was trained for 2,000 steps on 8 nodes (each node with 8xA100 80 GB GPUs).

Launch and setup: SFT

Evaluation#

Run inference using the AutoModel stack, followed by two scorers. MCQ accuracy is reported per class using regex and for open-ended Q&A (clip caption and player action description) we use LLM judge using the following prompt. Inference uses the generic AutoModel SFT config with two changes. Frames are capped at 32 to match training and we used max_new_tokens: 512 to allow generating the full caption.

MCQ scorer. The shared MCQ scorer in Evaluation is used as is.

LLM judge. The judge is used with two basketball modifications. In the VLM config avlm/evals/qa_llm_judge/vlm_configs/default.yaml, set metadata_mode: omit, and use the following prompt. We did not build a metadata field for basketball, while attaching the clip’s full annotation as metadata is an option to give the judge context to credit an answer that is correct but describes a different aspect of the clip than the reference. The full prompt the judge receives is shown below. The first paragraph is the shared default and the rest is the basketball system_prompt.

System message:

You are an expert VLM evaluator. Provide accurate, fair, and detailed scoring
based on the given criteria.

You are also an expert basketball analyst scoring answers about short basketball broadcast clips.
Two answer shapes occur. (1) clip_caption: the ground truth lists every logged action in
the fixed format "A player with jersey number <n> and jersey_color <colour> made a
<action> which result was made/miss and was assisted by other player with jersey number
<n>" (result clause only for shots, assist clause only for assisted made shots; actions
are 2PT Shot, 3PT Shot, Free Throw, Rebound, Foul, Turnover, Steal, Block, Violation).
Score factual agreement per action record: jersey number, jersey colour, action type,
made/miss result, and assist attribution all matter; missing or invented action records
are severe errors; wording and line order differences alone are not. (2)
player_action_description: the ground truth is a single sentence about one addressed
player; score whether the prediction attributes the SAME action (and made/miss outcome,
when stated) to the SAME player (jersey colour + number); extra correct detail is
harmless, a wrong player, wrong action, or wrong outcome is a failure. Do not reward
fluency.