8  Classify candidates

8.1 Make labeled cutouts

The block below makes 240 synthetic candidates of 21 by 21 pixels, in units of the sky noise. Half are labeled real, with a new source in the target. The other half simulate artifacts: a stationary star misregistered by one pixel between the images, or a cosmic ray (one bright pixel in the target). A genuinely moving source can produce the same dipole shape; its label depends on your science target, not on the shape alone.

import numpy as np

rng = np.random.default_rng(0)
n, size = 240, 21
yy, xx = np.mgrid[:size, :size] - size // 2


def psf(dx):  # a Gaussian point source, shifted dx pixels to the right
    return np.exp(-(yy**2 + (xx - dx) ** 2) / (2 * 1.5**2))


labels = np.tile([1, 0], n // 2)                          # 1 real, 0 bogus
kind = np.where(labels == 1, 0, rng.integers(1, 3, n))    # 0 new source, 1 misregistered star, 2 cosmic ray
peak = rng.uniform(5, 25, n)[:, None, None]
reference = rng.normal(0, 1, (n, size, size)).astype(np.float32)
target = rng.normal(0, 1, (n, size, size)).astype(np.float32)
target += np.where(kind[:, None, None] == 0, peak * psf(0), 0)
reference += np.where(kind[:, None, None] == 1, peak * psf(0), 0)
target += np.where(kind[:, None, None] == 1, peak * psf(1), 0)
target[kind == 2, size // 2, size // 2] += peak[kind == 2, 0, 0]
difference = target - reference
print(target.shape, target.dtype, "real:", labels.sum(), "bogus:", n - labels.sum())

(240, 21, 21) float32 real: 120 bogus: 120

Training and prediction use the cutouts as stored. They do not align, subtract or rescale them. Prepare every cutout the same way.

8.2 Write the dataset

xScan reads a dataset directory with one NumPy file per array (Table 8.1). It calls the target search and the reference template.

import json
from pathlib import Path

dataset = Path("sample/cutouts")
dataset.mkdir(parents=True, exist_ok=True)
split = np.repeat([0, 1, 2], [160, 40, 40])
for name, array in [("search", target), ("template", reference), ("difference", difference),
                    ("labels", labels), ("split", split)]:
    np.save(dataset / f"{name}.npy", array)
rows = [{"candidate_id": f"cand-{i}", "split_group": f"cand-{i}", "split": ("train", "val", "test")[s],
         "label": int(y), "label_source": "synthetic", "exposure_id": 0, "ccd_id": 0, "band": "r",
         "x": 10, "y": 10} for i, (s, y) in enumerate(zip(split, labels))]
(dataset / "metadata.jsonl").write_text("".join(json.dumps(row) + "\n" for row in rows))

42140

label_source records where a label came from. split_group names the object a candidate belongs to. Keep all candidates of one object in one split. The dataset check also requires exposure_id, ccd_id (the detector chip), band (the filter) and the cutout center x, y. Use placeholders when your data has none.

8.3 Check the dataset

validate_dataset_dir checks shapes, splits and metadata fields. The label guard, check_training_label_provenance, checks that the labels are binary, match the metadata, have a named source, cover both classes and keep each split_group in one split.

from cuphoton.xscan.dataset import validate_dataset_dir
from cuphoton.xscan.training import check_training_label_provenance

report = validate_dataset_dir(dataset)
print("valid:", report["valid"], "| input mode:", report["input_mode"])
print(report["splits"])
guard = check_training_label_provenance(dataset)
print("labels ok:", guard["ok"], guard["errors"])

valid: True | input mode: triplet
{'train': {'count': 160, 'positives': 80, 'negatives': 80}, 'val': {'count': 40, 'positives': 20, 'negatives': 20}, 'test': {'count': 40, 'positives': 20, 'negatives': 20}}
labels ok: True []

The command-line equivalents print the full JSON reports. With --require-ok, the label check exits with an error when it fails:

cuphoton xscan data-validate --dataset-dir sample/cutouts
cuphoton xscan data-check-training-labels --dataset-dir sample/cutouts --require-ok

Training runs the label guard again and stops when it fails.

8.4 Train a model

Training reads a YAML file. The model settings below make a tiny network that trains in seconds on a CPU. For the default network, which takes 51-pixel cutouts, start from the repository’s examples/xscan/train-triplet.example.yaml.

import shutil

Path("runs").mkdir(exist_ok=True)
Path("runs/classify-train.yaml").write_text("""\
dataset_dir: sample/cutouts
output_root: runs
run_name: classify-demo
epochs: 2
batch_size: 16
early_stopping_metric: val_loss   # keep the epoch with the lowest validation loss
device: auto                      # CUDA when PyTorch finds a GPU, otherwise the CPU
model:
  image_size: 21                  # cutout side in pixels
  depths: [1, 1]                  # two small encoder stages, for this demo only
  num_heads: [1, 1]
  embed_dims: [16, 16]
  decoder_embedding_dim: 16
""")
shutil.rmtree("runs/classify-demo", ignore_errors=True)  # xScan never writes into an existing run

Train the triplet model, which reads all three channels. train-inada-pair reads the target and reference only. The command prints a JSON summary. Save it to a file:

cuphoton xscan train-inada-triplet --config runs/classify-train.yaml > runs/classify-train.json

The run directory runs/classify-demo/ holds checkpoint.pt, the full configuration config.yaml, the per-epoch history.json and the run summary.json.

summary = json.loads(Path("runs/classify-train.json").read_text())
print("device:", summary["device"], "| saved epoch:", summary["early_stopping"]["best_epoch"],
      "| selected by:", summary["best_metric"]["name"])
for epoch in json.loads(Path("runs/classify-demo/history.json").read_text()):
    print(f"epoch {epoch['epoch']:.0f}: train loss {epoch['train_loss']:.3f}, "
          f"validation loss {epoch['val_loss']:.3f}, validation ROC AUC {epoch['val_roc_auc']:.3f}")

device: cpu | saved epoch: 2 | selected by: val_loss
epoch 1: train loss 0.531, validation loss 0.596, validation ROC AUC 1.000
epoch 2: train loss 0.069, validation loss 0.117, validation ROC AUC 1.000

The checkpoint holds the epoch with the best selection metric, which is not always the last epoch. When early_stopping_metric is unset, selection uses validation ROC AUC (defined in the next step), or validation loss for a single-class validation split. Early stopping remains disabled unless a metric or patience is set.

8.5 Evaluate on the test split

evaluate-real-bogus scores one split, test by default, and writes metrics, curves and per-candidate predictions to runs/classify-demo/evaluation/test/.

cuphoton xscan evaluate-real-bogus --run-dir runs/classify-demo --dataset-dir sample/cutouts > runs/classify-evaluate.json

evaluation = json.loads(Path("runs/classify-evaluate.json").read_text())
print(f"ROC AUC {evaluation['roc_auc']:.3f}, PR AUC {evaluation['pr_auc']:.3f}")
thresholds = evaluation["threshold_diagnostics"]
for name, result in [("fixed", thresholds["fixed_threshold"]),
                     ("validation-selected", thresholds["validation_selected"]["evaluated_split_metrics"])]:
    print(f"{name} threshold {result['threshold']:.3f}: accuracy {result['accuracy']:.3f}, "
          f"real missed {result['fn']}, bogus passed {result['fp']}")

ROC AUC 1.000, PR AUC 1.000
fixed threshold 0.500: accuracy 0.950, real missed 2, bogus passed 0
validation-selected threshold 0.500: accuracy 0.950, real missed 2, bogus passed 0

ROC AUC measures how often a random labeled-real candidate outranks a labeled-bogus one, counting ties as half. PR AUC is the trapezoidal area under the precision-recall curve. A candidate is classified as real when its score reaches the threshold. Next to the fixed threshold of 0.5, the command picks the threshold with the best validation accuracy and applies it to the test split. summary.md in the same directory is a readable version of the report.

These metrics describe this small synthetic test split. They do not establish survey performance or score calibration. On real data, choose a threshold on validation data for the missed-source and false-positive rates you can accept, then evaluate it once on an independent test split. Precision also depends on how common real sources are in the scored population.

8.6 Predict

infer-real-bogus only scores. It writes the logits and probabilities of one split to runs/classify-demo/inference/test/.

cuphoton xscan infer-real-bogus --run-dir runs/classify-demo --dataset-dir sample/cutouts > runs/classify-infer.json

probabilities = np.load("runs/classify-demo/inference/test/probabilities.npy")
print("p(real):", probabilities[:6].round(3))
print("labels: ", labels[split == 2][:6])

p(real): [0.909 0.065 0.933 0.039 0.925 0.326]
labels:  [1 0 1 0 1 0]

The arrays follow the order of the test split in the dataset.

8.7 Score your own cutouts in Python

For cutouts in memory, load the checkpoint and call the model. load_model_from_checkpoint rebuilds the network, loads the saved weights and sets evaluation mode. The triplet model takes a float32 tensor of shape (candidate, channel, y, x), with the channels target, reference, difference. It returns one logit per candidate. Its sigmoid is the model’s real-class score, exposed as probabilities by the inference commands.

import torch
from cuphoton.xscan.training import load_model_from_checkpoint

model, checkpoint, performance = load_model_from_checkpoint(Path("runs/classify-demo"), device=torch.device("cpu"))
cutouts = np.stack([target, reference, difference], axis=1)[split == 2][:6]
with torch.no_grad():
    logits = model(torch.from_numpy(cutouts))
print("p(real):", torch.sigmoid(logits).numpy().round(3))
print("same as infer-real-bogus:", np.allclose(torch.sigmoid(logits).numpy(), probabilities[:6]))

p(real): [0.909 0.065 0.933 0.039 0.925 0.326]
same as infer-real-bogus: True

Figure 8.1 shows test cutouts of each kind with these probabilities.

Plotting code

import matplotlib.pyplot as plt

test = np.flatnonzero(split == 2)
picks = [i for k in range(3) for i in test[kind[test] == k][:2]]
names = ["new source", "misregistered", "cosmic ray"]
fig, axes = plt.subplots(3, 6, figsize=(6.5, 3.8))
for col, i in enumerate(picks):
    top = max(target[i].max(), reference[i].max())
    for row, (image, cmap, lo, hi) in enumerate([(target[i], "magma", -3, top), (reference[i], "magma", -3, top),
                                                 (difference[i], "RdBu_r", -top / 2, top / 2)]):
        axes[row, col].imshow(image, origin="lower", cmap=cmap, vmin=lo, vmax=hi)
        axes[row, col].set_xticks([]); axes[row, col].set_yticks([])
    p = probabilities[np.searchsorted(test, i)]
    axes[0, col].set_title(f"{names[kind[i]]}\n$p$ = {p:.2f}", fontsize=8)
for row, label in enumerate(["Target", "Reference", "Difference"]):
    axes[row, 0].set_ylabel(label, fontsize=8)
fig.text(1, 1, "Synthetic data, for demonstration only", ha="right", va="bottom",
         fontsize=7, style="italic", color="0.4", transform=fig.transFigure)
fig.tight_layout()
plt.show()

8.8 Handle checkpoints

A checkpoint is a dictionary. model_config holds the network’s shape, model_state its weights, train_config the training configuration and xfit_feature_bundle the identity of optional xFit features.

print(sorted(checkpoint))
config = checkpoint["model_config"]
print(config["input_mode"], config["image_size"], config["embed_dims"])

['model_config', 'model_state', 'train_config', 'xfit_feature_bundle']
triplet 21 [16, 16]

load_model_from_checkpoint passes model_config to build_model and loads model_state into the result. Call build_model(**checkpoint["model_config"]) to build the same network in your own PyTorch code. A checkpoint only works on cutouts with its input mode, size and preprocessing. A wrong size fails late, inside the network. To reject a checkpoint before any model is built, pass a checkpoint_validator, a function that raises ValueError:

def require_triplet(size):
    def check(checkpoint):
        config = checkpoint["model_config"]
        if config["input_mode"] != "triplet" or config["image_size"] != size:
            raise ValueError(f"need a triplet model for {size} px cutouts, got "
                             f"{config['input_mode']} for {config['image_size']} px")
    return check


try:
    load_model_from_checkpoint(Path("runs/classify-demo"), checkpoint_validator=require_triplet(51))
except ValueError as error:
    print("ValueError:", error)

ValueError: need a triplet model for 51 px cutouts, got triplet for 21 px

8.9 Queue candidates for human review

Reviewers correct and extend labels through a review queue. review-queue ranks the predictions of one split. The default strategy, hybrid, favors candidates near the decision threshold most.

cuphoton xscan review-queue --run-dir runs/classify-demo --dataset-dir sample/cutouts --output-dir runs/classify-demo/review --max-items 8 > runs/classify-review.json

queue = Path("runs/classify-demo/review/queue.jsonl").read_text().splitlines()
for item in map(json.loads, queue[:4]):
    print(item["rank"], item["candidate_id"], f"p(real) {item['probability']:.2f}", "|", item["rank_reason"])

1 cand-214 p(real) 0.41 | uncertainty
2 cand-220 p(real) 0.36 | uncertainty
3 cand-206 p(real) 0.65 | uncertainty
4 cand-205 p(real) 0.33 | audit

The other review commands take the review directory (--review-dir) and follow Table 8.2.

The queue and the append-only annotations.jsonl are the review record. review-apply writes to a new --output-dir and leaves the source dataset unchanged.

8.10 Options

Table 8.3 lists the main settings of the training YAML file and the commands.

8.11 On a GPU

With the gpu extra, the same commands run on CUDA, which device: auto selects. GPU settings go in the performance section of the YAML file. They are off by default. On a CPU, xScan ignores mixed precision and pinned memory.

import yaml

config = yaml.safe_load(Path("runs/classify-train.yaml").read_text())
config.update(run_name="classify-gpu", device="cuda", performance={
    "amp_dtype": "bf16", "allow_tf32": True, "cudnn_benchmark": True, "num_workers": 2,
    "worker_start_method": "spawn", "pin_memory": True, "persistent_workers": True,
    "non_blocking_transfers": True})
Path("runs/classify-gpu.yaml").write_text(yaml.safe_dump(config, sort_keys=False))

cuphoton xscan train-inada-triplet --config runs/classify-gpu.yaml > runs/classify-gpu.json

For cutouts already on the GPU, predict_tensors scores a contiguous float32 tensor on an explicit CUDA device and returns logits and probabilities on it:

from cuphoton.xscan.training import predict_tensors

device = torch.device("cuda:0")
model, _, performance = load_model_from_checkpoint(Path("runs/classify-gpu"), device=device)
result = predict_tensors(model=model, images=torch.from_numpy(cutouts).to(device), device=device,
                         performance=performance)
print(result["probabilities"].cpu())

predict_fits cuts and scores cutouts straight from aligned FITS images on the GPU, without a dataset directory. It needs CuPy and an explicit CUDA device, and the cutout sides in stamp_shape must be odd. It does not align, subtract or normalize the images.

from cuphoton.xscan.fits_inference import FitsPlane, predict_fits

result = predict_fits(
    model=model, device=device, performance=performance,
    search=FitsPlane("target.fits", hdu=1), template=FitsPlane("reference-aligned.fits", hdu=1),
    difference=FitsPlane("difference.fits", hdu=1),
    centers_yx=[(200, 300), (450, 600)], candidate_ids=["candidate-a", "candidate-b"], stamp_shape=(21, 21),
)
print(result.candidate_ids, result.probabilities)

8.12 What is survey-specific

The array format, the model and the checkpoints are generic. These parts depend on the survey:

• Cutout preparation. Size, centering, scaling and masking are your choice. A checkpoint is valid only for cutouts prepared like its training set.
• Dataset builders. cuphoton xscan --help lists builders for several surveys’ products, such as data-build-autoscan. Example manifests are in examples/xscan/.
• Labels. Define which astrophysical changes count as real for your task. The label guard rejects placeholder labels from survey smoke datasets and survey object IDs that appear in two splits; it cannot verify that a human label is scientifically correct.
• Thresholds and calibration. Choose the threshold on a representative validation split, and check performance and score calibration on held-out data with the class balance and observing conditions you expect to encounter.

8.13 Troubleshooting

ERROR: [Errno 17] File exists: '.../runs/classify-demo'

Each training run needs a new directory. Change run_name or delete the old run.

ERROR: refusing to train with group leakage across train/validation/test splits, or refusing to train on labels that are not real-bogus usable: label_source_untrusted

The label guard failed. Run data-check-training-labels and fix the listed metadata. A label_source that is empty or contains “placeholder” or “unlabeled” is untrusted.

RuntimeError: The expanded size of the tensor (676) must match the existing size (121)

The cutouts differ in size from the checkpoint’s image_size. Use a checkpoint_validator to catch this at load time.

ValueError: triplet model expects 3-channel stacked input

Stack the channels target, reference and difference, in that order. A pair model takes the first two.

8.14 Next steps

• Add xFit dipole-fit features to the classifier: Chapter 7 and Optional xFit fusion.
• Classify every candidate of an image pair in one GPU pipeline: Chapter 9.
• Spread infer-real-bogus over many GPUs: Chapter 14.
• Full reference: the xScan component guide and the dataset contract.