A comparison is only as clear as its protocol.

One partition for every local training stage

All tasks share the same coda partition, keeping every locally trained representation separate from its test examples. Encoder training, scaler fitting, augmentation partners, auxiliary class weights, and linear-probe fitting are restricted to the training partition. This corrects the historical mismatch between encoder-training and identity-probe partitions.

Three random splits use seeds 0–2 and approximately 64/16/20% training/validation/test proportions, stratifying eligible identities and otherwise social unit. Recording-date splits use outer folds 0–2 of a five-fold stratified-group partition with shuffle seed 0, together with grouped validation partitions. Recording dates are session proxies.

Four additional evaluations hold out 2005, 2008, 2009, and 2010 in turn. The test year is completely excluded from training and validation. Validation uses a date-grouped fold from the remaining years; the neural seed is fixed at 0. A year result is one fit on one held-out population, not a repeated-seed mean.

Frozen representations, supervised probes

All encoder weights are frozen after training. Balanced logistic regression predicts social unit, coda type, or individual ID. Features are standardized on training data with fixed C = 1 and at most 2,000 iterations.

WhAM and AVES layers are selected per task on validation macro-F1 only, with earliest-layer tie breaking. BirdNET provides one representation. There is no training-plus-validation refit, and neural checkpoints use a fixed final epoch. The test scores are not used to choose the reported layer.

What macro-F1 measures here

Primary macro-F1 averages over true test classes observed in training, on supported examples. For each scored class, F1 is the harmonic mean of precision and recall. The unweighted mean of these class scores prevents the largest class from dominating the metric.

“Eligible” and “supported” are different. An identity can satisfy the dataset-wide eligibility rule but be absent from a particular training partition. Primary scoring removes those unsupported test examples. Full-eligible secondary scores retain unseen classes and are saved separately.

The grouped ID folds span three, six, and six scored identities. Random and grouped identity scores therefore describe different supported populations. A numerical gap between them is not a controlled estimate of recording-condition dependence alone.

For raw ICI, including unseen type labels changes the random/grouped type means from 0.987/0.956 to 0.968/0.919. Both views matter: supported-class scores describe the primary closed-set task, while full-eligible scores expose coverage limitations.

Repeated runs and uncertainty

The main tables report means and sample standard deviations over three random splits or three date-grouped folds. SD describes variation across the three runs; it is not a 95% confidence interval or a significance test. Date-grouped folds also differ in class composition and support.

Conditional date-cluster bootstrap intervals and paired contrasts are retained in the saved evidence. Their scope is the available test recordings and dates, not a new population. Some intervals cannot be estimated when fewer than two supported test dates are available. They should not be substituted for independent-population validation.

A study with an exploratory history

Earlier unit-balanced experiments informed the later comparisons. The matched uniform-sampling grid, year holdouts, frozen-input controls, and branch diagnostics were introduced in stages. Some controls followed partial main results. This is an exploratory study, not a preregistered confirmatory design.

All eight U/T/I settings were tested. The conference paper reports selected contrasts for space, while the ablation chapter and saved result tables retain the wider grid. Earlier training-regime scores are not pooled with the current matched ablations.

What local holdouts cannot establish

WhAM’s published training uses DSWP+CETI, and our evaluation is also drawn from DSWP. Exact checkpoint training membership is unavailable. Local train/test separation does not retroactively establish that a released foundation model never encountered these recordings, years, or populations during external pretraining.

The available cache spot checks address whether sampled feature vectors can be reproduced from the audio. They do not establish pretraining independence. Similarly, closed-set ID recognition does not demonstrate identification of previously unseen animals or the communicative meaning of a coda.

Evidence: protocol history, pretraining and cache provenance, saved predictions and splits, and numerical audit.