Sports Intelligence Playbooks

NVIDIA Sports Intelligence Playbooks give developers a structured framework to turn sports video, data, and domain expertise into specialized multimodal AI models. From data preparation and fine-tuning through evaluation and deployment, they help teams build AI that understands the game and powers a new generation of sports applications.

Sports Intelligence GitHub repository

What the Playbooks Provide

NVIDIA Sports Intelligence Playbooks provide developers with end-to-end recipes for taking their own sports footage, annotations, metadata, and domain knowledge and using them to customize multimodal models for sports understanding.

The lifecycle spans data acquisition, annotation and preparation, inference, evaluation, optimization, and deployment.

The playbooks bring together NVIDIA accelerated computing, open multimodal AI models, model customization, evaluation tools, and inference pipeline into a reference workflow developers can adapt to their sport and application.

The result is a path for turning general multimodal AI into specialized sports intelligence.

Playbook Includes

1. Complete sports intelligence starter kit. Covers the full lifecycle—from sports-video annotation and data preparation through SFT/LoRA training, inference, and evaluation. Its template-driven data pipeline systematically generates multiple training-example types, enabling models to be trained and evaluated across different aspects of sports understanding.

2. Video- and audio-first fine-tuning with sports-relevant customization. Provides tested recipes for sports video clips with audio, rather than relying primarily on simpler image-and-text examples available in public recipes. It also exposes input-resolution and frame-sampling settings that help preserve details important to sports analysis, such as player motion, ball trajectory, court geometry, and event timing.

3. Domain-specific, measurable evaluation. Includes per-question-class MCQ metrics and LLM-as-a-judge evaluation for captions and descriptive answers.

4. Reproducible, scalable execution. Supplies tested distributed workflows, checkpoint conversion and parity checks, and automated inference-to-evaluation pipelines.

5. Informed choice of training method and framework. Supports and compares full SFT and LoRA across NeMo AutoModel and Megatron-Bridge, helping users balance quality, memory, iteration speed, and checkpoint size. When properly tuned, Megatron-Bridge can deliver up to 3x higher training throughput than AutoModel.

6. Last but not least: shared practical learnings. Shares lessons from building, debugging, optimizing, and validating multimodal training workflows that can help others fine-tune multimodal language models for sports intelligence.

The playbooks cover the complete AVLM lifecycle—from collecting and annotating multimodal sports data through train/eval preparation, SFT/LoRA training, inference, evaluation, and deployment. Evaluation findings feed back into data and training choices for continuous improvement.

Playbooks Scripts: Implementation, launch scripts, and runbooks live in the Sports Intelligence Playbooks code repo under avlm/training/, avlm/inference/, and avlm/evals/.

Use the navigation on the left to browse by section.

Multimodal Language Models

• Data Curation
  • Example Tennis Annotation Guideline
  • From Tennis Annotation to Training Data
    • Pipeline Overview
    • What the Generators Do
    • Categorical Evaluation Mode
    • Train / Validation / Test Splits
    • Output Format
    • Question Templates (Summary)
    • Example Code and Documentation
    • Quick Start
• Training
  • Setup
    • Code Repo
    • Containers
    • Two Stacks (Pick One)
    • Shared Layout (Both Stacks)
    • Repo Bootstrap (Optional)
    • Megatron-Bridge Only
    • Interactive vs. Sbatch (Slurm)
    • Typical Workflow
  • SFT
    • What Gets Trained
    • Training Schedule and Batching
    • Optimizer and Learning Rate
    • Distributed Parallelism
    • Data, Video, and Sequence Budget
    • MoE Backend (AutoModel Only)
    • Weights & Biases
    • Quick Start
  • LoRA
    • SFT vs. LoRA (compute and checkpoint size)
    • What Gets Trained (Target Modules and Freeze)
    • Training Schedule and Batching
    • Optimizer and Learning Rate
    • Distributed Parallelism
    • Data, Video, and Sequence Budget
    • MoE Backend (AutoModel Only)
    • Checkpointing
    • Weights & Biases
    • Post-Training (Megatron-Bridge Only)
    • Quick Start
• Inference
  • Setup
    • Containers
    • Two Backends (Pick One)
    • Shared Layout
  • AutoModel Inference
  • Megatron-Bridge Inference
  • Inference Config Reference
    • Shared Fields (Both Backends)
    • CLI / Environment Overrides
    • AutoModel Fields
    • Megatron-Bridge Fields
  • LoRA Inference (Adapter-Only Weights)
    • AutoModel
    • Megatron-Bridge
  • Outputs
  • Quick Start
• Evaluation
  • Two Evaluation Modes
    • Tennis reference classes
  • MCQ Evaluation
    • Answer format
    • Results
    • Run MCQ eval
  • LLM-as-Judge Evaluation
    • Full event context for the judge
    • Judge prompt
    • Judge configuration
    • Run LLM judge
    • LLM judge aggregation
    • LLM judge HTML summary
  • Outputs Layout
  • Inference + Eval Pipeline
• Performance
  • Nomenclature
  • Fine-Tuning Performance
    • Best observed configurations
    • Example training time
  • Inference Performance
• Learnings
  • Data Size Impact
    • Expectations
    • Evidence from Our Tests
  • Frame Resolution Impact
    • Expectations
    • Evidence from Our Tests
  • Training Data Bias Impact
    • Expectations
    • Evidence from Our Tests
  • Video Chunking Impact
    • Expectations