Adapt the Vision Encoder for Fine-Grained Perception

Sports questions range from coarse to fine-grained. For example, recognizing jersey colors is coarse: the evidence is large and persists across frames. Reading a jersey number, judging whether a ball landed inside the line, or seeing who touched a loose ball is fine-grained: the evidence is small, moving, and often visible for only a few frames. Although all of this fine-grained visual information is initially captured at the raw pixel level, much of it is subsequently degraded or removed during downstream processing. Standard audio-visual language model (AVLM) architectures often compress visual inputs for efficiency—such as by downsampling, subsampling frames, or pooling spatial patches into fewer tokens—and rely on vision encoders pretrained for general video understanding. These processing steps favor coarse features, causing critical fine-grained details to be blended or discarded before reaching the backbone language model.

To improve fine-grained perception, unfreeze and adapt the vision encoder instead of fine-tuning the language model alone. This is to make the vision encoder trainable and maintain full temporal detail. Fine-tuning only the language model improves how visual evidence is used, while adapting the encoder improves what evidence reaches it.

Expectations

• Make the vision encoder trainable when the task needs details that pretraining may not be specialized for. A frozen encoder caps the accuracy that fine-tuning can reach. Training it lets the encoder allocate capacity to domain details such as jersey numbers, the ball, or line placement.

• Keep full detail when the evidence is brief or small. Some architectures merge consecutive frames or adjacent pixels into shared tokens, consequently, small details visible for only a few frames or across a few pixels get blended with their surroundings. Disabling such compression in the vision encoder, e.g., encoding each sampled frame on its own or turning off pixel binning, preserves more detail for the model to interpret.

• Expect the largest gains on perception-heavy questions. Player identity, fine event details, and structured captions that must list every participant can gain the most, while coarse questions near their ceiling may not move much. The changes to the vision encoder can stack. Validate each change and their combinations on held-out questions from your own use case, and choose the cheapest combination that meets the quality target.

• Plan for the added cost. Training the encoder adds training memory and compute. Disabling compression raises the number of visual tokens per frame, so at a fixed token budget you sample fewer frames per clip, or you pay for the extra tokens at both training and inference.

Evidence from Our Tests

In our experiments, we validated that a trainable vision encoder and per-frame encoding improve performance on fine-grained questions from held-out basketball games.

• Settings. Four SFT runs from the same base model: (1) Nemotron-3-Nano-Omni Baseline: Frozen vision encoder with two-frame merging, (2) No frame merging: Frozen vision encoder with per-frame encoding (no frame merging), (3) Trainable vision encoder: Trainable vision encoder with two-frame merging, and (4) Both changes: No frame merging + Trainable vision encoder.

• Controlled training. All four runs used the same training data, schedule, and per-clip visual-token budget. An unmerged frame results in twice as many tokens, so the runs without frame merging sampled half as many frames. The comparison therefore isolates the two settings rather than the token count.

• Evaluation. We evaluate on held-out samples from unseen games, balanced across question types. We report MCQ accuracy for the two jersey-number identification questions for the actor, such as “Which red-jersey player commits a foul in this video?”, and the assister, such as “The basket scored by the player wearing the green number 27 jersey was assisted. Which teammate made the assist?”. We also report the share of structured clip captions, such as “Analyze the basketball video and extract all player actions. For each action, identify the player’s jersey number, jersey colour, and action type.”, rated at least 9 out of 10 by an LLM judge.

Each modification provides independent performance gains, and combining both yields cumulative benefits. The largest gain is on clip captions, which must name every player in the clip, followed by the two jersey-number questions.

Figure 10 Held-out evaluation on unseen games. Unfreezing the vision encoder and retaining full temporal detail each improve accuracy on fine-grained questions, with the combination yielding the highest overall performance.