Executed revision notebook: saved outputs; no cells run by this page. Read the research record · All notebooks

V2 — corrected experiments and results

This notebook documents and loads the completed 72-run grid. It is separate from all original notebooks. The corrected runs used standalone scripts; the outputs below load their saved results rather than pretending the historical notebooks were rerun.

from pathlib import Path
import json, os, subprocess, sys
import pandas as pd
from IPython.display import display, Image, Markdown
ROOT = Path.cwd()
if not (ROOT / 'scripts_v2').is_dir():
    ROOT = ROOT.parent
assert (ROOT / 'scripts_v2').is_dir(), 'Open this notebook from the project or notebooks_v2 folder.'
RESULTS = ROOT / 'results_v2'
PYTHON = ROOT / '.venv' / 'bin' / 'python'
sys.path.insert(0, str(ROOT / 'scripts_v2'))
ENV = dict(os.environ, OPENBLAS_NUM_THREADS='1', OMP_NUM_THREADS='1')
def command(script, *args):
    result = subprocess.run([str(PYTHON), str(ROOT / 'scripts_v2' / script), *args],
                            cwd=ROOT, env=ENV, capture_output=True, text=True)
    print(result.stdout)
    if result.stderr:
        print(result.stderr)
    result.check_returncode()
print('Canonical results:', RESULTS)
print('Training is disabled by default; existing artifacts are read below.')

Canonical results: /Users/joaoquintanilha/Downloads/data_297_final_paper/results_v2
Training is disabled by default; existing artifacts are read below.

Optional reproduction

All switches default to False. Matching completed results are skipped by the runner; changing the experimental design requires a new output version. The recorded output alias results/review_revision_v1 resolves to results_v2 so original training hashes stay valid.

RUN_PREPROCESS = False
RUN_BASELINES = False
RUN_NEURAL = False
if RUN_PREPROCESS:
    command('revision_data.py')
if RUN_BASELINES:
    command('revision_experiments.py', '--stage', 'baselines')
if RUN_NEURAL:
    command('revision_experiments.py', '--stage', 'neural')
if not any([RUN_PREPROCESS, RUN_BASELINES, RUN_NEURAL]):
    print('No preprocessing or training launched.')

No preprocessing or training launched.

Completed grid

The grid contains 12 representations × 2 split protocols × 3 runs/folds. The two-epoch smoke run is excluded.

completion = json.loads((RESULTS / 'completion.json').read_text())
print(json.dumps(completion, indent=2))
assert completion['completed_runs'] == completion['expected_runs'] == 72
summary = pd.read_csv(RESULTS / 'summary.csv')
summary['mean ± sample SD'] = summary.apply(lambda r: f"{r['mean']:.3f} ± {r['sd']:.3f}", axis=1)
for protocol in ['random', 'date_grouped']:
    print(protocol)
    display(summary[summary.protocol == protocol].pivot(index='model', columns='task', values='mean ± sample SD'))

{
  "expected_runs": 72,
  "completed_runs": 72,
  "missing": []
}
random

task	coda_type	individual_id	unit
model
AVES	0.253 ± 0.028	0.491 ± 0.029	0.910 ± 0.010
AVES_Adapter	0.272 ± 0.027	0.479 ± 0.018	0.933 ± 0.009
AVES_ICI_Adapter	0.452 ± 0.023	0.582 ± 0.078	0.940 ± 0.009
BirdNET	0.215 ± 0.053	0.480 ± 0.022	0.887 ± 0.020
DCCE_ID	0.722 ± 0.051	0.588 ± 0.079	0.889 ± 0.004
DCCE_no_ID	0.800 ± 0.026	0.627 ± 0.057	0.907 ± 0.003
DCCE_swap_ID	0.636 ± 0.042	0.641 ± 0.066	0.879 ± 0.029
ICI_Temporal_Mel	0.804 ± 0.016	0.724 ± 0.039	0.837 ± 0.013
Raw_ICI	0.987 ± 0.009	0.439 ± 0.007	0.570 ± 0.023
Supervised_Dual	0.762 ± 0.119	0.684 ± 0.081	0.886 ± 0.017
Temporal_Mel	0.202 ± 0.025	0.429 ± 0.045	0.773 ± 0.007
WhAM	0.258 ± 0.046	0.450 ± 0.052	0.906 ± 0.011

date_grouped

task	coda_type	individual_id	unit
model
AVES	0.061 ± 0.022	0.074 ± 0.051	0.700 ± 0.100
AVES_Adapter	0.079 ± 0.029	0.059 ± 0.044	0.646 ± 0.134
AVES_ICI_Adapter	0.183 ± 0.037	0.081 ± 0.072	0.669 ± 0.119
BirdNET	0.063 ± 0.011	0.085 ± 0.022	0.612 ± 0.133
DCCE_ID	0.502 ± 0.018	0.257 ± 0.063	0.574 ± 0.074
DCCE_no_ID	0.616 ± 0.066	0.281 ± 0.109	0.687 ± 0.023
DCCE_swap_ID	0.476 ± 0.084	0.219 ± 0.089	0.611 ± 0.014
ICI_Temporal_Mel	0.668 ± 0.051	0.255 ± 0.062	0.608 ± 0.150
Raw_ICI	0.956 ± 0.036	0.379 ± 0.103	0.540 ± 0.036
Supervised_Dual	0.675 ± 0.058	0.270 ± 0.095	0.611 ± 0.080
Temporal_Mel	0.069 ± 0.012	0.130 ± 0.074	0.569 ± 0.094
WhAM	0.066 ± 0.016	0.101 ± 0.075	0.667 ± 0.155

Comparison

Primary macro-F1 uses true test classes seen in training. Candidate/support sets differ between ordinary and grouped identity tasks, so compare models within each protocol and consult support counts. Error bars are sample SD, not 95% confidence intervals.

display(Image(filename=str(RESULTS / 'figures' / 'comparison.png')))
display(pd.read_csv(RESULTS / 'split_support.csv'))

	protocol	seed	train	validation	test	id_train_classes	id_test_eligible	id_test_supported	id_test_supported_classes	unseen_ids
0	date_grouped	0	828	227	328	9	213	186	3	5719, 5720
1	date_grouped	1	901	211	271	7	121	111	6	5973, 6069
2	date_grouped	2	883	239	261	7	145	145	6	NaN
3	random	0	884	222	277	11	151	151	11	NaN
4	random	1	884	222	277	11	151	151	11	NaN
5	random	2	884	222	277	11	151	151	11	NaN

Paired comparisons and identity strata

The per-unit/type summaries subset the global classifier’s predictions; they do not refit within-unit classifiers. Very small or single-identity strata do not support broad conclusions.

display(pd.read_csv(RESULTS / 'paired_differences.csv'))
strata = pd.read_csv(RESULTS / 'identity_strata.csv')
display(strata[(strata.grouping == 'unit') & strata.model.isin(['DCCE_no_ID', 'DCCE_ID'])])

	protocol	model_a	model_b	task	paired_runs	mean_delta	sd_delta
0	random	DCCE_ID	DCCE_no_ID	unit	3	-0.017480	0.006837
1	random	DCCE_ID	DCCE_no_ID	coda_type	3	-0.077994	0.038518
2	random	DCCE_ID	DCCE_no_ID	individual_id	3	-0.038928	0.043129
3	random	DCCE_swap_ID	DCCE_ID	unit	3	-0.010135	0.025492
4	random	DCCE_swap_ID	DCCE_ID	coda_type	3	-0.085534	0.074021
5	random	DCCE_swap_ID	DCCE_ID	individual_id	3	0.052794	0.095301
6	random	DCCE_ID	Supervised_Dual	unit	3	0.003534	0.014468
7	random	DCCE_ID	Supervised_Dual	coda_type	3	-0.039788	0.081399
8	random	DCCE_ID	Supervised_Dual	individual_id	3	-0.095826	0.039724
9	random	DCCE_ID	ICI_Temporal_Mel	unit	3	0.051940	0.010066
10	random	DCCE_ID	ICI_Temporal_Mel	coda_type	3	-0.082260	0.064305
11	random	DCCE_ID	ICI_Temporal_Mel	individual_id	3	-0.135778	0.073042
12	random	DCCE_ID	AVES_ICI_Adapter	unit	3	-0.050629	0.007266
13	random	DCCE_ID	AVES_ICI_Adapter	coda_type	3	0.270426	0.067840
14	random	DCCE_ID	AVES_ICI_Adapter	individual_id	3	0.005932	0.015594
15	date_grouped	DCCE_ID	DCCE_no_ID	unit	3	-0.112680	0.090799
16	date_grouped	DCCE_ID	DCCE_no_ID	coda_type	3	-0.113409	0.048259
17	date_grouped	DCCE_ID	DCCE_no_ID	individual_id	3	-0.023920	0.057532
18	date_grouped	DCCE_swap_ID	DCCE_ID	unit	3	0.037218	0.064539
19	date_grouped	DCCE_swap_ID	DCCE_ID	coda_type	3	-0.026341	0.089267
20	date_grouped	DCCE_swap_ID	DCCE_ID	individual_id	3	-0.037998	0.119123
21	date_grouped	DCCE_ID	Supervised_Dual	unit	3	-0.037581	0.145719
22	date_grouped	DCCE_ID	Supervised_Dual	coda_type	3	-0.173084	0.075422
23	date_grouped	DCCE_ID	Supervised_Dual	individual_id	3	-0.012491	0.074419
24	date_grouped	DCCE_ID	ICI_Temporal_Mel	unit	3	-0.033691	0.076842
25	date_grouped	DCCE_ID	ICI_Temporal_Mel	coda_type	3	-0.165256	0.066301
26	date_grouped	DCCE_ID	ICI_Temporal_Mel	individual_id	3	0.001915	0.017167
27	date_grouped	DCCE_ID	AVES_ICI_Adapter	unit	3	-0.094710	0.059657
28	date_grouped	DCCE_ID	AVES_ICI_Adapter	coda_type	3	0.319098	0.054103
29	date_grouped	DCCE_ID	AVES_ICI_Adapter	individual_id	3	0.176067	0.025556

	model	protocol	seed	grouping	stratum	n	n_true_id_classes	macro_f1	accuracy	caution
60	DCCE_ID	date_grouped	0	unit	F	186	3	0.197058	0.155914	Subset of global predictions; no within-unit r...
75	DCCE_no_ID	date_grouped	0	unit	F	186	3	0.215309	0.209677	Subset of global predictions; no within-unit r...
244	DCCE_ID	date_grouped	1	unit	F	111	6	0.322351	0.297297	Subset of global predictions; no within-unit r...
260	DCCE_no_ID	date_grouped	1	unit	F	111	6	0.406426	0.405405	Subset of global predictions; no within-unit r...
436	DCCE_ID	date_grouped	2	unit	F	145	6	0.252342	0.365517	Subset of global predictions; no within-unit r...
452	DCCE_no_ID	date_grouped	2	unit	F	145	6	0.221774	0.268966	Subset of global predictions; no within-unit r...
680	DCCE_ID	random	0	unit	A	5	2	1.000000	1.000000	Subset of global predictions; no within-unit r...
681	DCCE_ID	random	0	unit	D	2	2	1.000000	1.000000	Subset of global predictions; no within-unit r...
682	DCCE_ID	random	0	unit	F	144	7	0.574243	0.687500	Subset of global predictions; no within-unit r...
709	DCCE_no_ID	random	0	unit	A	5	2	1.000000	1.000000	Subset of global predictions; no within-unit r...
710	DCCE_no_ID	random	0	unit	D	2	2	0.333333	0.500000	Subset of global predictions; no within-unit r...
711	DCCE_no_ID	random	0	unit	F	144	7	0.664860	0.736111	Subset of global predictions; no within-unit r...
1016	DCCE_ID	random	1	unit	A	5	2	0.761905	0.800000	Subset of global predictions; no within-unit r...
1017	DCCE_ID	random	1	unit	D	2	2	0.333333	0.500000	Subset of global predictions; no within-unit r...
1018	DCCE_ID	random	1	unit	F	144	7	0.629429	0.666667	Subset of global predictions; no within-unit r...
1042	DCCE_no_ID	random	1	unit	A	5	2	1.000000	1.000000	Subset of global predictions; no within-unit r...
1043	DCCE_no_ID	random	1	unit	D	2	2	1.000000	1.000000	Subset of global predictions; no within-unit r...
1044	DCCE_no_ID	random	1	unit	F	144	7	0.628698	0.659722	Subset of global predictions; no within-unit r...
1336	DCCE_ID	random	2	unit	A	5	2	1.000000	1.000000	Subset of global predictions; no within-unit r...
1337	DCCE_ID	random	2	unit	D	2	2	1.000000	1.000000	Subset of global predictions; no within-unit r...
1338	DCCE_ID	random	2	unit	F	144	7	0.440120	0.590278	Subset of global predictions; no within-unit r...
1364	DCCE_no_ID	random	2	unit	A	5	2	1.000000	1.000000	Subset of global predictions; no within-unit r...
1365	DCCE_no_ID	random	2	unit	D	2	2	0.500000	0.500000	Subset of global predictions; no within-unit r...
1366	DCCE_no_ID	random	2	unit	F	144	7	0.562493	0.638889	Subset of global predictions; no within-unit r...

Corrected recording-year diagnostic

All dates are retained. Layer profiles use validation scores and are descriptive across correlated layers, not causal evidence.

display(Image(filename=str(RESULTS / 'diagnostics' / 'corrected_year_diagnostic.png')))

Independent verification

Recompute reported metrics from saved predictions; check training membership, source/input hashes, scalers, and the 50-epoch histories.

command('verify_revision_runs.py', '--require-complete')

{
  "checked_runs": 72,
  "checked_neural_runs": 36,
  "complete": true,
  "probe_convergence_warnings": 0,
  "verified": [
    "feature hashes and row alignment",
    "disjoint splits and recording groups",
    "training-only coda membership and ICI scalers",
    "50 finite-loss epochs",
    "saved embeddings and checkpoints",
    "prediction labels and eligibility",
    "recomputed F1 and accuracy",
    "validation-argmax layer choice",
    "source snapshot and input provenance"
  ]
}

Interpretation

The original large ID-supervision gain is not reproduced. The simple ICI + temporal-mel floor is a strong ordinary-split ID baseline, and recording-date separation reduces performance for the tested audio-based models. These results support a more cautious empirical comparison, not blanket superiority of the contrastive architecture. Read reports_v2/revision_results.md, revision_response_map.md, and revision_manuscript_changes.md before updating the paper. Full label-free/single-channel/bandwidth ablations and external-population validation remain future work.