Evaluation#

AVLM evaluation pipeline

The evaluation pipeline routes inference predictions by evaluation_type: MCQ answers use deterministic letter matching, while open-ended answers use an LLM judge. The resulting scores are aggregated into JSON metrics and an HTML report.

After inference writes predictions, we score them with two complementary methods:

  • MCQ accuracy — letter matching for multiple-choice questions (fast, deterministic)

  • LLM-as-judge — rubric scoring for open-ended captions and descriptions where wording varies but factual alignment matters

Both approaches apply across sports. Code lives in the Sports Intelligence Playbooks repository under avlm/evals/; the shipped tennis question classes and prompts are a reference implementation, and the same patterns extend to other domains with sport-specific templates, metadata schemas, and judge rubrics. Predictions must exist first at:

avlm/inference/outputs/<INFERENCE_NAME>/predictions.jsonl

Each line is one (video, question) pair. See Inference for generating predictions.

Two Evaluation Modes#

Routing uses the evaluation_type field on each row in predictions.jsonl:

  • mcq — scored locally by letter matching; no external API calls

  • open_ended — scored by an LLM judge; requires a judge API key

The class field stores the template name (for example server_winner or point_caption). Set both fields during data preparation when defining templates (see Data Curation). MCQ answers use (A) ... through (D) ...; open-ended templates use free-text ground truth.

Tennis reference classes#

Tennis MCQ classes include match facts such as score_after, server_winner, how_point_ended, and num_shots_exchanged, plus categorical visual and reasoning classes. Open-ended classes include point_caption, how_point_ended_description, and audio_cues. Categorical rows also carry super_category and fine_category labels for breakdowns across visual recognition, commentary, match facts, rules/reasoning, and audio. See avlm/data_prep_example/tennis/categorical_evaluation/CATEGORICAL_EVAL.MD for the current taxonomy.

MCQ Evaluation#

The MCQ scorer reads inference predictions and compares each model answer to the annotated correct letter. The headline metric is the percentage of correct answers.

Answer format#

Answers must start with (A) through (D); optional Markdown bold is accepted. The scorer extracts the leading letter from model output and ground truth. Malformed reference answers are skipped with a warning. Unparseable model outputs count as num_parse_failed and remain in the denominator, lowering accuracy.

Results#

Results are written to mcq/mcq_results.json under the inference output directory. It reports pct_correct (a 0–1 fraction), total_mcq_questions, correct/incorrect counts, parse and inference failures, and the same metrics under per_class.

Run MCQ eval#

INFERENCE_DIR=avlm/inference/outputs/<INFERENCE_NAME> \
bash avlm/evals/scripts/run_mcq_eval.sh

Output:

avlm/inference/outputs/<INFERENCE_NAME>/mcq/mcq_results.json

LLM-as-Judge Evaluation#

The LLM judge scores open_ended rows. A separate model rates each answer against the reference on a 1–10 scale and returns a short justification plus notes on factual alignment.

Use LLM judging when there is no single letter or exact string to match — for example, event captions or play-by-play descriptions where phrasing can differ but facts must hold.

Full event context for the judge#

The judge does not see only the reference answer and the model reply. For each example it also receives:

  1. Reference answer — the annotation from the eval set

  2. Original prompt — the full question the VLM was asked (including the video reference)

  3. Model answer — the generated text (inference separates any internal reasoning trace before writing predictions)

  4. Event metadata — the full annotated row for that clip from preprocessed source JSON

Why include event metadata?#

Open-ended ground truth is often a short caption or play-by-play string. The model answer may use different wording but still be correct — or it may add extra factual claims that are not spelled out in the reference (for example, whether the point ended on an ace, a double fault, or a winner, or whether a serve was first or second).

Comparing prediction to ground truth alone is not enough in those cases:

  • Ground truth alone cannot confirm or refute every fine-grained claim the model makes.

  • Strict pred-vs-GT matching would unfairly penalize valid extra detail that the annotator simply did not repeat in the short reference string.

  • Lenient semantic matching would let plausible but wrong details slip through.

Event metadata breaks that tradeoff. It is the structured annotation row for that point — the same fields used during data curation (see Data Curation): server/receiver, scores, how_point_ended, num_serving_attempts_until_successful, rally length, captions, and related labels. The judge treats metadata as authoritative context to:

  • Verify specific claims in the prediction (for example, cross-check serve-related statements against num_serving_attempts_until_successful).

  • Disambiguate point-ending categories (ace, double fault, winner, unforced error, and so on) when the reference caption is vague or abbreviated.

  • Avoid false hallucination penalties when the model adds a detail that ground truth omitted but metadata supports.

  • Penalize contradictions when the prediction conflicts with structured fields, even if the wording sounds plausible.

If metadata is omitted, the rubric falls back to strict pred-vs-GT scoring and penalizes unsupported additions. When metadata is present, scoring is primarily grounded-truth alignment, with metadata used as a secondary consistency check.

How metadata is attached#

During data preparation, open-ended rows can be augmented with structured point metadata. Inference copies that metadata field into predictions.jsonl; the judge does not load a separate annotation directory. metadata_mode in avlm/evals/qa_llm_judge/vlm_configs/default.yaml controls whether metadata is placed in the user or system message, selected automatically, or omitted. The shipped config uses the user message.

Judge prompt#

The rubric lives in avlm/evals/utils/prompt.py. At runtime the scorer fills in the reference answer, original prompt, model answer, and event metadata. An optional domain-specific paragraph from system_prompt in avlm/evals/qa_llm_judge/vlm_configs/default.yaml is appended to the system message.

System message (default):

You are an expert VLM evaluator. Provide accurate, fair, and detailed scoring
based on the given criteria.

You are also an expert tennis commentator and rules adjudicator. Interpret tennis terminology precisely
(serve, return, rally, winner, forced/unforced error, ace, double fault, volley, down-the-line, cross-court, baseline).
When metadata includes score flow or serve attempts, penalize only direct contradictions;
do not overrule matching GT/pred serve-sequence descriptions from ambiguous attempt-count fields.
Prefer concise, technical justifications.

User message (template; REPLACE_* slots are filled per example):

Your task is to compare the model prediction directly against the conversation
and ground truth, and use the provided metadata as authoritative context for
verification and disambiguation. Prioritize agreement with gt, and use metadata
only for direct contradictions or clear disambiguation. Do not invent
contradictions from ambiguous metadata fields. Treat unambiguous player aliases
as the same player (for example, Amritraj and Riske-Amritraj) when the
conversation or gt makes the mapping clear. For first serve/double fault/ace
serve-related claims, use metadata fields like
'num_serving_attempts_until_successful' as supporting context, but do not use
that field to penalize a fault/let/then-successful-serve sequence when gt and
pred agree on that sequence and metadata does not explicitly state a different
complete serve sequence. If the prediction adds a specific point-ending
category (like Ace, Double Fault, Winner, etc) that is not mentioned in the gt,
do NOT penalize unless this addition is directly contradicted by the metadata.
If metadata supports or is consistent with the pred's extra detail, do not
treat it as a hallucination. However, if pred adds a point-ending type that is
directly contradicted by metadata, penalize accordingly.

If no metadata is provided, score purely on pred vs gt and be strict about
extra unsupported details.

IMPORTANT: Respond only with valid JSON. The "score" field must be an integer
from 1 to 10, where 10 is best, never 0.

**Ground Truth**
REPLACE_GT

**Conversation**
REPLACE_CONVERSATION

**Prediction**
REPLACE_PREDICTION

**Metadata**
REPLACE_METADATA

**Evaluation Guidance:**
- Primary: Similarity to gt (semantic and factual)
- Secondary: Consistency with metadata when present; penalize claims directly
  contradicted by available metadata
- **MCQ / single-letter answers**: If gt is of the form "(A) ...", "(B) ...",
  "(C) ...", "(D) ..." and pred is only the letter (e.g. "A", "B", "C", "D")
  or "A." / "The answer is A.", treat as a **full match (score 10 ** when the
  letter matches the gt choice. Do not penalize for missing the full "(X) ..."
  text.
- Treat unambiguous player aliases as equivalent, including
  surname/full-name/hyphenated variants such as "Amritraj" and
  "Riske-Amritraj"
- For serve/double fault/ace claims, check metadata fields like
  'num_serving_attempts_until_successful' as supporting context; do not let that
  field override matching gt/pred fault/let/then-successful-serve descriptions
  unless metadata explicitly gives a different complete serve sequence
- If pred adds a point-ending type (Ace, Double Fault, etc) not in gt, check
  metadata:
    - If metadata supports or is consistent, do NOT penalize
    - If metadata directly contradicts, penalize
    - If metadata is absent, be strict and penalize unsupported additions
- Ignore plausible but unsupported details that are not grounded in gt/metadata

**Scoring (1-10, integers):**
- 10: Pred essentially matches gt; no contradictions with metadata
- 9: Very strong match; at most tiny omissions
- 8: Minor omissions or wording differences; consistent with metadata
- 7: Captures main idea; some errors/unsupported details
- 6: Partially relevant; noticeable errors/contradictions
- 1-5: Poor match or contradicted by metadata

Respond ONLY with a JSON object of the form:
{
  "score": <1-10>,
  "reasoning": "<brief, precise justification>",
  "strengths": ["<strength 1>", "<strength 2>"],
  "weaknesses": ["<weakness 1>", "<weakness 2>"],
  "metadata_alignment": "<note on consistency with metadata>"
}

The tennis examples in the prompt (serve attempts, point-ending categories) illustrate how metadata cross-check works; adapt prompt.py and the config system_prompt for other sports.

Judge configuration#

Copy and edit avlm/evals/qa_llm_judge/vlm_configs/default.yaml. You need:

  • An OpenAI-compatible API base URL and judge model name

Optional settings cover parallelism, timeouts, metadata placement, output fields, and domain-specific system guidance.

Set the API key before running:

export CLIENT_API_KEY=<YOUR_CLIENT_API_KEY>

LLM judge requires openai, pyyaml, and tqdm. Install them in base Python or a venv; if using a venv, set JUDGE_VENV_PATH to its root. Run eval commands from the repository root; the shell wrappers accept configuration through environment variables rather than positional arguments.

Run LLM judge#

INFERENCE_DIR=avlm/inference/outputs/<INFERENCE_NAME> \
bash avlm/evals/scripts/run_qa_llm_judge.sh

With config override and sample limit:

VLM_SCORER_CONFIG=avlm/evals/qa_llm_judge/vlm_configs/default.yaml \
MAX_LLM_JUDGE_SAMPLES=100 \
INFERENCE_DIR=avlm/inference/outputs/<INFERENCE_NAME> \
bash avlm/evals/scripts/run_qa_llm_judge.sh

Output:

avlm/inference/outputs/<INFERENCE_NAME>/llm_judge/llm_judge_predictions.json

This file contains statistics, per-example results, failures, error counts, and run metadata.

LLM judge aggregation#

A second step aggregates judge output overall and by class:

INFERENCE_DIR=avlm/inference/outputs/<INFERENCE_NAME> \
bash avlm/evals/scripts/run_qa_llm_judge_eval.sh

Output:

avlm/inference/outputs/<INFERENCE_NAME>/llm_judge/llm_judge_results.json

The result includes success/failure counts, mean and median scores, range, score distribution, and per_class metrics.

LLM judge HTML summary#

Generate a browsable report from the per-example judge output:

INFERENCE_DIR=avlm/inference/outputs/<INFERENCE_NAME> \
bash avlm/evals/scripts/run_qa_llm_judge_summary.sh

Output:

avlm/inference/outputs/<INFERENCE_NAME>/llm_judge/llm_judge_summary.html

Outputs Layout#

avlm/inference/outputs/<INFERENCE_NAME>/
├── predictions.jsonl          # from inference
├── mcq/
│   └── mcq_results.json       # MCQ accuracy
└── llm_judge/
    ├── llm_judge_predictions.json   # per-entry judge scores and failures
    ├── llm_judge_results.json       # aggregated judge metrics
    └── llm_judge_summary.html       # browsable report

Inference + Eval Pipeline#

avlm/inference/common/scripts/run_inference_eval_pipeline.sh runs inference and evaluation end to end. The default stages are inference,mcq,judge,judge_eval,judge_summary. Set INFERENCE_BACKEND to automodel, megatron_bridge, or megatron_core.

INFERENCE_BACKEND=automodel \
INFERENCE_CONFIG=avlm/inference/automodel/configs/default_sft.yaml \
bash avlm/inference/common/scripts/run_inference_eval_pipeline.sh

Score existing predictions only:

STAGES=mcq,judge,judge_eval,judge_summary \
INFERENCE_DIR=avlm/inference/outputs/<INFERENCE_NAME> \
bash avlm/inference/common/scripts/run_inference_eval_pipeline.sh

See avlm/inference/README.md for pipeline overrides and eval-suite examples, and avlm/evals/README.MD for script-level reference.