Complete review revision: V3¶
This notebook reads the preserved experiment outputs. It does not rerun training by default. V3 is exploratory because V2 results were examined before its design. See reports_v3/protocol.md for the fixed experiment grid.
from pathlib import Path
import json
import pandas as pd
pd.set_option('display.max_rows', 120)
from IPython.display import display, Markdown, Image
ROOT = Path.cwd()
if not (ROOT / 'results_v3').exists():
ROOT = ROOT.parent
assert (ROOT / 'results_v3').exists()
OUT = ROOT / 'results_v3'
print(json.loads((OUT / 'completion.json').read_text()))
{'completed_v2': 72, 'completed_v3': 114, 'expected_v3': 114, 'completed_v3_main': 114, 'completed_year_holdout': 24, 'completed_new_total': 168, 'expected_new_total': 168, 'completed_feature_fusion': 30, 'primary_evaluations_including_v2': 240, 'diagnostic_branch_probes': 24}
display(Markdown((ROOT / 'reports_v3/supervision_matrix.md').read_text()))
V3 supervision and implementation matrix¶
U/T/I refer to labels used during representation training. Every row later freezes the representation and trains a separate supervised linear probe. All V3 samplers are uniform; auxiliary class weights use training labels only.
| Model | U | T | I | Input | Pairing | Representation | Dimension |
|---|---|---|---|---|---|---|---|
| Dual_U0T0I0 | 0 | 0 | 0 | dual, 8 kHz | same-coda augmented views | trained, 50 epochs | 64 |
| Dual_U0T0I1 | 0 | 0 | 1 | dual, 8 kHz | same-coda augmented views | trained, 50 epochs | 64 |
| Dual_U0T1I0 | 0 | 1 | 0 | dual, 8 kHz | same-coda augmented views | trained, 50 epochs | 64 |
| Dual_U0T1I1 | 0 | 1 | 1 | dual, 8 kHz | same-coda augmented views | trained, 50 epochs | 64 |
| Dual_U1T0I0 | 1 | 0 | 0 | dual, 8 kHz | same-unit whole codas | trained, 50 epochs | 64 |
| Dual_U1T0I1 | 1 | 0 | 1 | dual, 8 kHz | same-unit whole codas | trained, 50 epochs | 64 |
| Dual_U1T1I0 | 1 | 1 | 0 | dual, 8 kHz | same-unit whole codas | trained, 50 epochs | 64 |
| Dual_U1T1I1 | 1 | 1 | 1 | dual, 8 kHz | same-unit whole codas | trained, 50 epochs | 64 |
| rhythm_UTI0 | 1 | 1 | 0 | rhythm, 8 kHz | same-unit whole codas | trained, 50 epochs | 64 |
| rhythm_UTI1 | 1 | 1 | 1 | rhythm, 8 kHz | same-unit whole codas | trained, 50 epochs | 64 |
| spectral_UTI0 | 1 | 1 | 0 | spectral, 8 kHz | same-unit whole codas | trained, 50 epochs | 64 |
| spectral_UTI1 | 1 | 1 | 1 | spectral, 8 kHz | same-unit whole codas | trained, 50 epochs | 64 |
| concat_UTI0 | 1 | 1 | 0 | concat, 8 kHz | same-unit whole codas | trained, 50 epochs | 128 |
| concat_UTI1 | 1 | 1 | 1 | concat, 8 kHz | same-unit whole codas | trained, 50 epochs | 128 |
| Dual_swap_UTI | 1 | 1 | 1 | dual, 8 kHz | same-unit swapped channels | trained, 50 epochs | 64 |
| Dual_16k_UTI0 | 1 | 1 | 0 | dual, 16 kHz | same-unit whole codas | trained, 50 epochs | 64 |
| Dual_16k_UTI1 | 1 | 1 | 1 | dual, 16 kHz | same-unit whole codas | trained, 50 epochs | 64 |
| AVES_FT | CE | CE | CE | audio | none | transformer fine-tuned, 10 epochs; frontend frozen | 64 |
| AVES_ICI_FT | CE | CE | CE | audio + ICI | none | transformer fine-tuned, 10 epochs; frontend frozen | 64 |
Single-channel auxiliary heads both use the available channel. Dual/concatenation type heads use rhythm and ID heads use spectrum. U=0 removes unit labels from representation training but not from evaluation partitioning/probes. Label-free objectives still use annotation-derived ICI inputs. AVES FT has a different pretraining history, parameter count and epoch budget from DCCE.
V2 benchmark: downstream whale-label access¶
These columns describe local downstream training, not all labels used in external pretraining. Every row uses a subsequently trained supervised probe.
| Model | Unit use | Type loss | ID loss | Unit-balanced sampler | Representation | Input |
|---|---|---|---|---|---|---|
| Raw ICI | none | no | no | no | fixed features | annotated ICI |
| Temporal mel | none | no | no | no | fixed features | audio |
| ICI + temporal mel | none | no | no | no | fixed features | annotated ICI + audio |
| WhAM | none | no | no | no | frozen pretrained; WhAM/AVES layer selected on validation | audio |
| AVES | none | no | no | no | frozen pretrained; WhAM/AVES layer selected on validation | audio |
| BirdNET | none | no | no | no | frozen pretrained; WhAM/AVES layer selected on validation | audio |
| DCCE no ID | whole-coda unit pairing | yes | no | yes | trained, 50 epochs | annotated ICI + audio |
| DCCE ID | whole-coda unit pairing | yes | yes | yes | trained, 50 epochs | annotated ICI + audio |
| DCCE swap ID | same-unit swapped channels | yes | yes | yes | trained, 50 epochs | annotated ICI + audio |
| Supervised dual | unit CE | yes | yes | yes | trained head/fusion; AVES backbone frozen in adapter rows | annotated ICI + audio |
| AVES adapter | unit CE | yes | yes | yes | trained head/fusion; AVES backbone frozen in adapter rows | audio |
| AVES + ICI adapter | unit CE | yes | yes | yes | trained head/fusion; AVES backbone frozen in adapter rows | annotated ICI + audio |
Additional frozen input-matched controls¶
WhAM + ICI, AVES + ICI, and BirdNET + ICI concatenate annotated timing with frozen audio embeddings. They use no local representation-training labels, losses, pairing, or sampler. Only the training-only scaler and supervised linear probe are fitted; WhAM/AVES layers are selected on validation data. External pretraining label/data access remains as described above. These controls are evaluated on all six random/date partitions and all four year holdouts.
The strict year diagnostic reuses the same definitions for Raw ICI, ICI + temporal mel, WhAM, AVES, DCCE 110 and DCCE 111. The latter two are trained afresh on each year partition with uniform sampling. Saved-branch probes are post-hoc diagnostics on fixed DCCE 110/111 checkpoints and do not replace the primary fused-representation results.
summary = pd.read_csv(OUT / 'summary.csv')
display(summary.pivot_table(index=['version','protocol','model'], columns='task', values=['mean','std','count']))
| count | mean | std | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| task | coda_type | individual_id | unit | coda_type | individual_id | unit | coda_type | individual_id | unit | ||
| version | protocol | model | |||||||||
| v2 | date_grouped | AVES | 3.0 | 3.0 | 3.0 | 0.061377 | 0.073839 | 0.699606 | 0.022273 | 0.051010 | 0.099810 |
| AVES_Adapter | 3.0 | 3.0 | 3.0 | 0.078889 | 0.058777 | 0.646327 | 0.028765 | 0.043898 | 0.134228 | ||
| AVES_ICI_Adapter | 3.0 | 3.0 | 3.0 | 0.183220 | 0.081183 | 0.668603 | 0.037160 | 0.072343 | 0.118781 | ||
| BirdNET | 3.0 | 3.0 | 3.0 | 0.062522 | 0.084921 | 0.611727 | 0.010718 | 0.021797 | 0.133217 | ||
| DCCE_ID | 3.0 | 3.0 | 3.0 | 0.502319 | 0.257250 | 0.573893 | 0.017592 | 0.062791 | 0.074437 | ||
| DCCE_no_ID | 3.0 | 3.0 | 3.0 | 0.615727 | 0.281170 | 0.686573 | 0.065519 | 0.108523 | 0.023382 | ||
| DCCE_swap_ID | 3.0 | 3.0 | 3.0 | 0.475978 | 0.219252 | 0.611110 | 0.084046 | 0.088875 | 0.014209 | ||
| ICI_Temporal_Mel | 3.0 | 3.0 | 3.0 | 0.667575 | 0.255334 | 0.607584 | 0.050841 | 0.061528 | 0.149813 | ||
| Raw_ICI | 3.0 | 3.0 | 3.0 | 0.956408 | 0.379035 | 0.539629 | 0.036222 | 0.102940 | 0.035826 | ||
| Supervised_Dual | 3.0 | 3.0 | 3.0 | 0.675403 | 0.269741 | 0.611473 | 0.057831 | 0.095472 | 0.079680 | ||
| Temporal_Mel | 3.0 | 3.0 | 3.0 | 0.068907 | 0.129634 | 0.568908 | 0.011673 | 0.074012 | 0.094056 | ||
| WhAM | 3.0 | 3.0 | 3.0 | 0.065956 | 0.100725 | 0.666666 | 0.016308 | 0.074519 | 0.155343 | ||
| random | AVES | 3.0 | 3.0 | 3.0 | 0.252601 | 0.491131 | 0.909781 | 0.027528 | 0.029407 | 0.009811 | |
| AVES_Adapter | 3.0 | 3.0 | 3.0 | 0.271576 | 0.478530 | 0.932544 | 0.027186 | 0.018312 | 0.009488 | ||
| AVES_ICI_Adapter | 3.0 | 3.0 | 3.0 | 0.451531 | 0.582073 | 0.939995 | 0.023057 | 0.078058 | 0.009122 | ||
| BirdNET | 3.0 | 3.0 | 3.0 | 0.215446 | 0.480356 | 0.887335 | 0.052697 | 0.021991 | 0.019882 | ||
| DCCE_ID | 3.0 | 3.0 | 3.0 | 0.721958 | 0.588005 | 0.889366 | 0.051336 | 0.079361 | 0.004248 | ||
| DCCE_no_ID | 3.0 | 3.0 | 3.0 | 0.799952 | 0.626933 | 0.906846 | 0.026307 | 0.056997 | 0.002644 | ||
| DCCE_swap_ID | 3.0 | 3.0 | 3.0 | 0.636423 | 0.640799 | 0.879231 | 0.042360 | 0.065771 | 0.028993 | ||
| ICI_Temporal_Mel | 3.0 | 3.0 | 3.0 | 0.804218 | 0.723783 | 0.837426 | 0.015689 | 0.039093 | 0.012893 | ||
| Raw_ICI | 3.0 | 3.0 | 3.0 | 0.987453 | 0.439054 | 0.570300 | 0.008672 | 0.007079 | 0.023350 | ||
| Supervised_Dual | 3.0 | 3.0 | 3.0 | 0.761745 | 0.683831 | 0.885832 | 0.119211 | 0.080852 | 0.016999 | ||
| Temporal_Mel | 3.0 | 3.0 | 3.0 | 0.201753 | 0.429238 | 0.772509 | 0.024982 | 0.044984 | 0.006913 | ||
| WhAM | 3.0 | 3.0 | 3.0 | 0.258139 | 0.449791 | 0.905812 | 0.045638 | 0.051887 | 0.011270 | ||
| v3 | date_grouped | AVES_FT | 3.0 | 3.0 | 3.0 | 0.071606 | 0.059693 | 0.626429 | 0.050473 | 0.026242 | 0.157076 |
| AVES_ICI_FT | 3.0 | 3.0 | 3.0 | 0.210730 | 0.072674 | 0.640972 | 0.068920 | 0.038355 | 0.157738 | ||
| Dual_16k_UTI0 | 3.0 | 3.0 | 3.0 | 0.708279 | 0.249994 | 0.624067 | 0.146885 | 0.131308 | 0.003592 | ||
| Dual_16k_UTI1 | 3.0 | 3.0 | 3.0 | 0.669489 | 0.172909 | 0.611894 | 0.011272 | 0.141528 | 0.055992 | ||
| Dual_U0T0I0 | 3.0 | 3.0 | 3.0 | 0.468995 | 0.181433 | 0.541650 | 0.042615 | 0.133170 | 0.090488 | ||
| Dual_U0T0I1 | 3.0 | 3.0 | 3.0 | 0.441014 | 0.158994 | 0.484759 | 0.055972 | 0.054353 | 0.101975 | ||
| Dual_U0T1I0 | 3.0 | 3.0 | 3.0 | 0.662280 | 0.208878 | 0.551268 | 0.080695 | 0.090873 | 0.078475 | ||
| Dual_U0T1I1 | 3.0 | 3.0 | 3.0 | 0.679337 | 0.173883 | 0.551426 | 0.036979 | 0.068378 | 0.113046 | ||
| Dual_U1T0I0 | 3.0 | 3.0 | 3.0 | 0.348014 | 0.244719 | 0.517787 | 0.039266 | 0.084209 | 0.139986 | ||
| Dual_U1T0I1 | 3.0 | 3.0 | 3.0 | 0.270115 | 0.133899 | 0.511206 | 0.010946 | 0.038532 | 0.117560 | ||
| Dual_U1T1I0 | 3.0 | 3.0 | 3.0 | 0.692072 | 0.248122 | 0.597502 | 0.088280 | 0.129935 | 0.077931 | ||
| Dual_U1T1I1 | 3.0 | 3.0 | 3.0 | 0.627668 | 0.223143 | 0.595997 | 0.074429 | 0.175119 | 0.052135 | ||
| Dual_swap_UTI | 3.0 | 3.0 | 3.0 | 0.471134 | 0.189278 | 0.621251 | 0.098548 | 0.069411 | 0.070424 | ||
| concat_UTI0 | 3.0 | 3.0 | 3.0 | 0.859337 | 0.275378 | 0.549133 | 0.014456 | 0.116570 | 0.171126 | ||
| concat_UTI1 | 3.0 | 3.0 | 3.0 | 0.849676 | 0.239112 | 0.527221 | 0.082756 | 0.042940 | 0.140180 | ||
| rhythm_UTI0 | 3.0 | 3.0 | 3.0 | 0.927080 | 0.352297 | 0.486565 | 0.046769 | 0.133690 | 0.090828 | ||
| rhythm_UTI1 | 3.0 | 3.0 | 3.0 | 0.918903 | 0.330671 | 0.505141 | 0.044746 | 0.143114 | 0.113215 | ||
| spectral_UTI0 | 3.0 | 3.0 | 3.0 | 0.062041 | 0.053608 | 0.540588 | 0.025225 | 0.046902 | 0.129682 | ||
| spectral_UTI1 | 3.0 | 3.0 | 3.0 | 0.062850 | 0.091767 | 0.558569 | 0.011001 | 0.016134 | 0.093290 | ||
| random | AVES_FT | 3.0 | 3.0 | 3.0 | 0.243197 | 0.414024 | 0.923969 | 0.025432 | 0.078094 | 0.006931 | |
| AVES_ICI_FT | 3.0 | 3.0 | 3.0 | 0.466130 | 0.543899 | 0.912700 | 0.058863 | 0.016969 | 0.010566 | ||
| Dual_16k_UTI0 | 3.0 | 3.0 | 3.0 | 0.911201 | 0.630982 | 0.901471 | 0.057115 | 0.046188 | 0.006359 | ||
| Dual_16k_UTI1 | 3.0 | 3.0 | 3.0 | 0.727462 | 0.623031 | 0.893116 | 0.088185 | 0.045712 | 0.007781 | ||
| Dual_U0T0I0 | 3.0 | 3.0 | 3.0 | 0.582662 | 0.623958 | 0.693250 | 0.041934 | 0.064456 | 0.026777 | ||
| Dual_U0T0I1 | 3.0 | 3.0 | 3.0 | 0.565726 | 0.592438 | 0.673675 | 0.016813 | 0.061273 | 0.015982 | ||
| Dual_U0T1I0 | 3.0 | 3.0 | 3.0 | 0.769836 | 0.591471 | 0.709490 | 0.085769 | 0.057232 | 0.026989 | ||
| Dual_U0T1I1 | 3.0 | 3.0 | 3.0 | 0.801521 | 0.600153 | 0.725360 | 0.045655 | 0.080143 | 0.013393 | ||
| Dual_U1T0I0 | 3.0 | 3.0 | 3.0 | 0.492673 | 0.434126 | 0.861146 | 0.090645 | 0.066791 | 0.001385 | ||
| Dual_U1T0I1 | 3.0 | 3.0 | 3.0 | 0.348301 | 0.474068 | 0.885048 | 0.111685 | 0.031556 | 0.018935 | ||
| Dual_U1T1I0 | 3.0 | 3.0 | 3.0 | 0.865400 | 0.597034 | 0.900959 | 0.114735 | 0.051078 | 0.020301 | ||
| Dual_U1T1I1 | 3.0 | 3.0 | 3.0 | 0.735336 | 0.580833 | 0.907371 | 0.090526 | 0.011902 | 0.010447 | ||
| Dual_swap_UTI | 3.0 | 3.0 | 3.0 | 0.708329 | 0.657122 | 0.853837 | 0.073161 | 0.066612 | 0.012703 | ||
| concat_UTI0 | 3.0 | 3.0 | 3.0 | 0.938746 | 0.635740 | 0.875769 | 0.062073 | 0.015360 | 0.024578 | ||
| concat_UTI1 | 3.0 | 3.0 | 3.0 | 0.926869 | 0.695642 | 0.872877 | 0.059922 | 0.054608 | 0.010124 | ||
| rhythm_UTI0 | 3.0 | 3.0 | 3.0 | 0.978318 | 0.477465 | 0.581279 | 0.020882 | 0.036741 | 0.043923 | ||
| rhythm_UTI1 | 3.0 | 3.0 | 3.0 | 0.973165 | 0.454129 | 0.604256 | 0.012880 | 0.060516 | 0.052289 | ||
| spectral_UTI0 | 3.0 | 3.0 | 3.0 | 0.202073 | 0.357716 | 0.875258 | 0.021276 | 0.103299 | 0.022978 | ||
| spectral_UTI1 | 3.0 | 3.0 | 3.0 | 0.199069 | 0.444400 | 0.853682 | 0.037031 | 0.092184 | 0.006178 | ||
Conditional uncertainty¶
Date-cluster bootstrap confidence intervals condition on each fitted model. They do not capture training variability. Intervals are not estimated for fewer than two supported test dates. Classes absent from a bootstrap resample score zero under the fixed original class set. Three-run SD and per-run intervals answer different questions. Full intervals are in results_v3/cluster_bootstrap.csv; secondary scores including unseen classes are in results_v3/secondary_summary_including_unseen.csv.
intervals = pd.read_csv(OUT / 'cluster_bootstrap.csv')
display(intervals)
| model | protocol | seed | version | task | lower | upper | n_dates | n_test | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | AVES | date_grouped | 0 | v2 | unit | 0.282067 | 0.620450 | 5 | 328 |
| 1 | AVES | date_grouped | 0 | v2 | coda_type | 0.007123 | 0.047641 | 5 | 326 |
| 2 | AVES | date_grouped | 0 | v2 | individual_id | NaN | NaN | 1 | 186 |
| 3 | AVES_Adapter | date_grouped | 0 | v2 | unit | 0.235757 | 0.548072 | 5 | 328 |
| 4 | AVES_Adapter | date_grouped | 0 | v2 | coda_type | 0.011496 | 0.057891 | 5 | 326 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 553 | spectral_UTI0 | random | 2 | v3 | coda_type | 0.101263 | 0.224341 | 40 | 277 |
| 554 | spectral_UTI0 | random | 2 | v3 | individual_id | 0.138896 | 0.340289 | 23 | 151 |
| 555 | spectral_UTI1 | random | 2 | v3 | unit | 0.746105 | 0.904120 | 40 | 277 |
| 556 | spectral_UTI1 | random | 2 | v3 | coda_type | 0.097461 | 0.197164 | 40 | 277 |
| 557 | spectral_UTI1 | random | 2 | v3 | individual_id | 0.163604 | 0.449678 | 23 | 151 |
558 rows × 9 columns
within = pd.read_csv(OUT / 'within_unit_id.csv')
display(within)
print('Blank scores indicate fewer than two supported train/test identities.')
| model | protocol | seed | version | unit | n_train | n_test | n_train_ids | n_test_ids | macro_f1 | selected_layer | n_validation | validation_macro_f1 | probe_convergence_warning | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | AVES | date_grouped | 0 | v2 | A | 0 | 0 | 0 | 0 | NaN | NaN | NaN | NaN | NaN |
| 1 | AVES | date_grouped | 0 | v2 | D | 10 | 0 | 2 | 0 | NaN | NaN | NaN | NaN | NaN |
| 2 | AVES | date_grouped | 0 | v2 | F | 420 | 186 | 7 | 3 | 0.019608 | 10.0 | 111.0 | 0.257803 | False |
| 3 | AVES_Adapter | date_grouped | 0 | v2 | A | 0 | 0 | 0 | 0 | NaN | NaN | NaN | NaN | NaN |
| 4 | AVES_Adapter | date_grouped | 0 | v2 | D | 10 | 0 | 2 | 0 | NaN | NaN | NaN | NaN | NaN |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 553 | spectral_UTI0 | random | 2 | v3 | D | 6 | 2 | 2 | 2 | 0.333333 | 0.0 | 2.0 | 1.000000 | False |
| 554 | spectral_UTI0 | random | 2 | v3 | F | 458 | 144 | 7 | 7 | 0.343738 | 0.0 | 115.0 | 0.379803 | False |
| 555 | spectral_UTI1 | random | 2 | v3 | A | 18 | 5 | 2 | 2 | 0.583333 | 0.0 | 4.0 | 0.733333 | False |
| 556 | spectral_UTI1 | random | 2 | v3 | D | 6 | 2 | 2 | 2 | 0.333333 | 0.0 | 2.0 | 0.000000 | False |
| 557 | spectral_UTI1 | random | 2 | v3 | F | 458 | 144 | 7 | 7 | 0.370712 | 0.0 | 115.0 | 0.411015 | False |
558 rows × 14 columns
Blank scores indicate fewer than two supported train/test identities.
for protocol in ['random','date_grouped']:
figure = OUT / 'figures' / f'{protocol}.png'
if figure.exists():
display(Image(filename=str(figure)))
verification = OUT / 'verification.json'
if verification.exists():
display(json.loads(verification.read_text()))
else:
print('Full verification is pending completion of the experiment grid.')
preserved = ROOT / 'registry_v3/v2_preservation.json'
display(json.loads(preserved.read_text()))
{'checked': 114,
'expected': 114,
'missing': [],
'all_complete': True,
'checks': 'source and split hashes; train/test separation; train-only scaler; uniform sampler metadata; fixed epochs; finite losses and embeddings; supported labels and recomputed scores; probe convergence'}
{'checked': 516, 'changed': []}
year_report = ROOT / 'reports_v3/year_holdout.md'
if year_report.exists():
display(Markdown(year_report.read_text()))
display(json.loads((OUT / 'year_holdout_verification.json').read_text()))
else:
print('Year-holdout diagnostic is queued after the main grid.')
Explicit year holdout diagnostic¶
This exploratory amendment was specified after partial V3 random-split results were visible, before any year-holdout execution. Each year is held out completely; training and validation use disjoint dates from other years. All six prespecified models and all four years are reported. Neural seed is fixed at 0, so these are four different populations, not four repeat-seed estimates.
Partition sizes¶
| Held-out year | Train codas | Validation codas | Test codas |
|---|---|---|---|
| 2005 | 687 | 328 | 368 |
| 2008 | 845 | 275 | 263 |
| 2009 | 967 | 321 | 95 |
| 2010 | 570 | 156 | 657 |
unit¶
| Model | 2005 | 2008 | 2009 | 2010 |
|---|---|---|---|---|
| Raw ICI | 0.224 (368/368 supported) | 0.425 (263/263 supported) | 0.541 (95/95 supported) | 0.476 (657/657 supported) |
| ICI + temporal mel | 0.649 (368/368 supported) | 0.360 (263/263 supported) | 0.778 (95/95 supported) | 0.327 (657/657 supported) |
| WhAM | 0.822 (368/368 supported) | 0.371 (263/263 supported) | 0.789 (95/95 supported) | 0.422 (657/657 supported) |
| AVES | 0.750 (368/368 supported) | 0.510 (263/263 supported) | 0.833 (95/95 supported) | 0.377 (657/657 supported) |
| DCCE 110 | 0.592 (368/368 supported) | 0.362 (263/263 supported) | 0.767 (95/95 supported) | 0.346 (657/657 supported) |
| DCCE 111 | 0.448 (368/368 supported) | 0.391 (263/263 supported) | 0.713 (95/95 supported) | 0.357 (657/657 supported) |
coda_type¶
| Model | 2005 | 2008 | 2009 | 2010 |
|---|---|---|---|---|
| Raw ICI | 0.957 (358/368 supported) | 0.908 (263/263 supported) | 1.000 (95/95 supported) | 0.965 (636/657 supported) |
| ICI + temporal mel | 0.589 (358/368 supported) | 0.607 (263/263 supported) | 0.977 (95/95 supported) | 0.427 (636/657 supported) |
| WhAM | 0.077 (358/368 supported) | 0.045 (263/263 supported) | 0.166 (95/95 supported) | 0.051 (636/657 supported) |
| AVES | 0.065 (358/368 supported) | 0.047 (263/263 supported) | 0.103 (95/95 supported) | 0.029 (636/657 supported) |
| DCCE 110 | 0.596 (358/368 supported) | 0.702 (263/263 supported) | 1.000 (95/95 supported) | 0.476 (636/657 supported) |
| DCCE 111 | 0.583 (358/368 supported) | 0.546 (263/263 supported) | 0.839 (95/95 supported) | 0.421 (636/657 supported) |
individual_id¶
| Model | 2005 | 2008 | 2009 | 2010 |
|---|---|---|---|---|
| Raw ICI | 0.404 (236/314 supported) | 0.542 (57/57 supported) | 0.412 (37/64 supported) | 0.232 (292/319 supported) |
| ICI + temporal mel | 0.353 (236/314 supported) | 0.491 (57/57 supported) | 0.410 (37/64 supported) | 0.128 (292/319 supported) |
| WhAM | 0.238 (236/314 supported) | 0.229 (57/57 supported) | 0.038 (37/64 supported) | 0.093 (292/319 supported) |
| AVES | 0.193 (236/314 supported) | 0.207 (57/57 supported) | 0.228 (37/64 supported) | 0.068 (292/319 supported) |
| DCCE 110 | 0.258 (236/314 supported) | 0.607 (57/57 supported) | 0.468 (37/64 supported) | 0.197 (292/319 supported) |
| DCCE 111 | 0.257 (236/314 supported) | 0.670 (57/57 supported) | 0.349 (37/64 supported) | 0.186 (292/319 supported) |
Full-eligible macro-F1 values (including unseen labels) are in year_holdout_metrics.csv. Conditional date-cluster intervals are in year_holdout_intervals.csv. Class support and composition differ across years. These tests remove shared years between train/validation and test, but performance differences still combine biological, recording and label-composition changes; they do not identify a causal recording artifact.
{'checked': 24, 'expected': 24, 'all_complete': True}
for report in ['feature_fusion','branch_probes']:
path = ROOT / 'reports_v3' / f'{report}.md'
if path.exists():
display(Markdown(path.read_text()))
Input-matched frozen-feature controls¶
Exploratory addition: all three frozen pretrained families receive the same annotated ICI through direct concatenation. Only the downstream linear probe is trained. Multi-layer selection uses validation data exclusively. These controls match available inputs; they do not match external pretraining history or local auxiliary-label objectives.
date_grouped¶
| Model | Unit | Type | ID |
|---|---|---|---|
| AVES_ICI | 0.712 ± 0.111 | 0.366 ± 0.084 | 0.128 ± 0.103 |
| BirdNET_ICI | 0.628 ± 0.095 | 0.287 ± 0.061 | 0.100 ± 0.055 |
| WhAM_ICI | 0.768 ± 0.057 | 0.376 ± 0.151 | 0.153 ± 0.135 |
random¶
| Model | Unit | Type | ID |
|---|---|---|---|
| AVES_ICI | 0.926 ± 0.011 | 0.667 ± 0.044 | 0.652 ± 0.044 |
| BirdNET_ICI | 0.884 ± 0.018 | 0.497 ± 0.028 | 0.580 ± 0.031 |
| WhAM_ICI | 0.921 ± 0.005 | 0.722 ± 0.027 | 0.598 ± 0.028 |
Year holdouts¶
| Model | Year | Unit | Type | ID |
|---|---|---|---|---|
| AVES_ICI | 2005 | 0.675 | 0.440 | 0.263 |
| AVES_ICI | 2008 | 0.498 | 0.332 | 0.330 |
| AVES_ICI | 2009 | 0.835 | 0.733 | 0.376 |
| AVES_ICI | 2010 | 0.354 | 0.303 | 0.057 |
| BirdNET_ICI | 2005 | 0.646 | 0.238 | 0.229 |
| BirdNET_ICI | 2008 | 0.355 | 0.239 | 0.195 |
| BirdNET_ICI | 2009 | 0.819 | 0.718 | 0.126 |
| BirdNET_ICI | 2010 | 0.323 | 0.330 | 0.101 |
| WhAM_ICI | 2005 | 0.773 | 0.406 | 0.305 |
| WhAM_ICI | 2008 | 0.410 | 0.367 | 0.459 |
| WhAM_ICI | 2009 | 0.856 | 0.961 | 0.255 |
| WhAM_ICI | 2010 | 0.451 | 0.382 | 0.146 |
All per-run support counts and secondary scores are saved in feature_fusion_metrics.csv. Year holdouts control local probe training, not unknown historical pretraining exposure.
Descriptive channel probes¶
Post-hoc analysis of 12 fixed checkpoints. Encoders are not retrained. Every channel uses the same supervised, training-only frozen-probe protocol as fusion. Re-extracted fusion embeddings are checked against saved embeddings. These are diagnostic scores, not replacements selected for the primary result.
date_grouped: Dual_U1T1I0¶
| Output | Unit | Type | ID |
|---|---|---|---|
| rhythm | 0.558 ± 0.091 | 0.917 ± 0.065 | 0.325 ± 0.144 |
| spectral | 0.516 ± 0.149 | 0.057 ± 0.023 | 0.098 ± 0.069 |
| fusion | 0.598 ± 0.078 | 0.692 ± 0.088 | 0.248 ± 0.130 |
date_grouped: Dual_U1T1I1¶
| Output | Unit | Type | ID |
|---|---|---|---|
| rhythm | 0.552 ± 0.101 | 0.911 ± 0.071 | 0.320 ± 0.150 |
| spectral | 0.511 ± 0.125 | 0.055 ± 0.032 | 0.074 ± 0.045 |
| fusion | 0.596 ± 0.052 | 0.628 ± 0.074 | 0.223 ± 0.175 |
random: Dual_U1T1I0¶
| Output | Unit | Type | ID |
|---|---|---|---|
| rhythm | 0.639 ± 0.027 | 0.983 ± 0.023 | 0.459 ± 0.052 |
| spectral | 0.863 ± 0.013 | 0.158 ± 0.023 | 0.367 ± 0.116 |
| fusion | 0.901 ± 0.020 | 0.865 ± 0.115 | 0.597 ± 0.051 |
random: Dual_U1T1I1¶
| Output | Unit | Type | ID |
|---|---|---|---|
| rhythm | 0.632 ± 0.028 | 0.982 ± 0.019 | 0.461 ± 0.054 |
| spectral | 0.861 ± 0.001 | 0.169 ± 0.021 | 0.443 ± 0.059 |
| fusion | 0.907 ± 0.010 | 0.735 ± 0.091 | 0.581 ± 0.012 |
Reproduction¶
Run scripts_v3/experiments.py for the 102 DCCE/control evaluations and scripts_v3/aves_finetune.py for the 12 pretrained-transformer adaptations. scripts_v3/year_holdout.py adds the 24 strict year-holdout evaluations. Runners verify existing source hashes before skipping completed outputs. Analysis scripts regenerate tables and confidence intervals. Consult README_v3.md; training is intentionally not an automatic notebook side effect.