{ "cells": [ { "cell_type": "markdown", "id": "6201d45b", "metadata": {}, "source": [ "# V2 — data audit and shared splits\n", "\n", "New notebook for the corrected revision. Original notebooks remain unchanged. These cells inspect the completed artifacts and run read-only regression checks; they do not regenerate features or train models." ] }, { "cell_type": "code", "execution_count": 1, "id": "97d1fdc4", "metadata": { "execution": { "iopub.execute_input": "2026-09-20T16:41:42.901353Z", "iopub.status.busy": "2026-09-20T16:41:42.901145Z", "iopub.status.idle": "2026-09-20T16:41:43.324434Z", "shell.execute_reply": "2026-09-20T16:41:43.324010Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Canonical results: /Users/joaoquintanilha/Downloads/data_297_final_paper/results_v2\n", "Training is disabled by default; existing artifacts are read below.\n" ] } ], "source": [ "from pathlib import Path\n", "import json, os, subprocess, sys\n", "import pandas as pd\n", "from IPython.display import display, Image, Markdown\n", "ROOT = Path.cwd()\n", "if not (ROOT / 'scripts_v2').is_dir():\n", " ROOT = ROOT.parent\n", "assert (ROOT / 'scripts_v2').is_dir(), 'Open this notebook from the project or notebooks_v2 folder.'\n", "RESULTS = ROOT / 'results_v2'\n", "PYTHON = ROOT / '.venv' / 'bin' / 'python'\n", "sys.path.insert(0, str(ROOT / 'scripts_v2'))\n", "ENV = dict(os.environ, OPENBLAS_NUM_THREADS='1', OMP_NUM_THREADS='1')\n", "def command(script, *args):\n", " result = subprocess.run([str(PYTHON), str(ROOT / 'scripts_v2' / script), *args],\n", " cwd=ROOT, env=ENV, capture_output=True, text=True)\n", " print(result.stdout)\n", " if result.stderr:\n", " print(result.stderr)\n", " result.check_returncode()\n", "print('Canonical results:', RESULTS)\n", "print('Training is disabled by default; existing artifacts are read below.')\n" ] }, { "cell_type": "markdown", "id": "727f6b91", "metadata": {}, "source": [ "## Data eligibility\n", "\n", "Ambiguous IDs and the singleton are excluded only from identity training/scoring. All 1,383 clean codas remain available for unit/type tasks." ] }, { "cell_type": "code", "execution_count": 2, "id": "1d7e247e", "metadata": { "execution": { "iopub.execute_input": "2026-09-20T16:41:43.325918Z", "iopub.status.busy": "2026-09-20T16:41:43.325747Z", "iopub.status.idle": "2026-09-20T16:41:43.337368Z", "shell.execute_reply": "2026-09-20T16:41:43.336937Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Clean codas: 1383\n", "Eligible identity codas: 754\n", "Identity classes: 11\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
codas
eligible754
unknown620
ambiguous8
singleton1
\n", "
" ], "text/plain": [ " codas\n", "eligible 754\n", "unknown 620\n", "ambiguous 8\n", "singleton 1" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "audit = json.loads((RESULTS / 'data_audit.json').read_text())\n", "print('Clean codas:', audit['clean'])\n", "print('Eligible identity codas:', audit['id_eligible'])\n", "print('Identity classes:', len(audit['classes']['individual_id']))\n", "display(pd.Series(audit['id_exclusions'], name='codas').to_frame())" ] }, { "cell_type": "markdown", "id": "85b1c5bf", "metadata": {}, "source": [ "## Corrected year counts\n", "\n", "Explicit day/month/year parsing retains all records. Association motivates controlled evaluation; it does not prove that models rely on recording artifacts." ] }, { "cell_type": "code", "execution_count": 3, "id": "3ad26bbd", "metadata": { "execution": { "iopub.execute_input": "2026-09-20T16:41:43.338580Z", "iopub.status.busy": "2026-09-20T16:41:43.338461Z", "iopub.status.idle": "2026-09-20T16:41:43.345768Z", "shell.execute_reply": "2026-09-20T16:41:43.345295Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ADF
year
2005100358
20082717165
2009301847
2010174132351
\n", "
" ], "text/plain": [ " A D F\n", "year \n", "2005 10 0 358\n", "2008 27 171 65\n", "2009 30 18 47\n", "2010 174 132 351" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Chi-squared: 530.0435927400224 df: 6\n", "Cramér’s V: 0.43775358015833493\n" ] } ], "source": [ "display(pd.read_csv(RESULTS / 'year_unit_counts.csv', index_col=0))\n", "print('Chi-squared:', audit['chi_squared'], 'df:', audit['chi_squared_df'])\n", "print(\"Cramér’s V:\", audit['cramers_v'])" ] }, { "cell_type": "markdown", "id": "3462892d", "metadata": {}, "source": [ "## Shared split membership and identity support\n", "\n", "Three ordinary splits and the first three distinct folds of one fixed five-fold recording-date partition. A date is a session proxy. Every task inherits the same coda membership. Unseen test identities are reported explicitly." ] }, { "cell_type": "code", "execution_count": 4, "id": "5ac7e605", "metadata": { "execution": { "iopub.execute_input": "2026-09-20T16:41:43.347010Z", "iopub.status.busy": "2026-09-20T16:41:43.346888Z", "iopub.status.idle": "2026-09-20T16:41:43.356121Z", "shell.execute_reply": "2026-09-20T16:41:43.355578Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
protocolseedtrainvalidationtestid_train_classesid_test_eligibleid_test_supportedid_test_supported_classesunseen_ids
0date_grouped0828227328921318635719, 5720
1date_grouped1901211271712111165973, 6069
2date_grouped288323926171451456NaN
3random08842222771115115111NaN
4random18842222771115115111NaN
5random28842222771115115111NaN
\n", "
" ], "text/plain": [ " protocol seed train validation test id_train_classes \\\n", "0 date_grouped 0 828 227 328 9 \n", "1 date_grouped 1 901 211 271 7 \n", "2 date_grouped 2 883 239 261 7 \n", "3 random 0 884 222 277 11 \n", "4 random 1 884 222 277 11 \n", "5 random 2 884 222 277 11 \n", "\n", " id_test_eligible id_test_supported id_test_supported_classes unseen_ids \n", "0 213 186 3 5719, 5720 \n", "1 121 111 6 5973, 6069 \n", "2 145 145 6 NaN \n", "3 151 151 11 NaN \n", "4 151 151 11 NaN \n", "5 151 151 11 NaN " ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
protocolseedtaskn_test_eligiblen_test_supported
0date_grouped0unit328328
1date_grouped0coda_type328326
2date_grouped0individual_id213186
3date_grouped1unit271271
4date_grouped1coda_type271271
5date_grouped1individual_id121111
6date_grouped2unit261261
7date_grouped2coda_type261261
8date_grouped2individual_id145145
9random0unit277277
10random0coda_type277276
11random0individual_id151151
12random1unit277277
13random1coda_type277277
14random1individual_id151151
15random2unit277277
16random2coda_type277277
17random2individual_id151151
\n", "
" ], "text/plain": [ " protocol seed task n_test_eligible n_test_supported\n", "0 date_grouped 0 unit 328 328\n", "1 date_grouped 0 coda_type 328 326\n", "2 date_grouped 0 individual_id 213 186\n", "3 date_grouped 1 unit 271 271\n", "4 date_grouped 1 coda_type 271 271\n", "5 date_grouped 1 individual_id 121 111\n", "6 date_grouped 2 unit 261 261\n", "7 date_grouped 2 coda_type 261 261\n", "8 date_grouped 2 individual_id 145 145\n", "9 random 0 unit 277 277\n", "10 random 0 coda_type 277 276\n", "11 random 0 individual_id 151 151\n", "12 random 1 unit 277 277\n", "13 random 1 coda_type 277 277\n", "14 random 1 individual_id 151 151\n", "15 random 2 unit 277 277\n", "16 random 2 coda_type 277 277\n", "17 random 2 individual_id 151 151" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "display(pd.read_csv(RESULTS / 'split_support.csv'))\n", "display(pd.read_csv(RESULTS / 'task_support.csv'))" ] }, { "cell_type": "markdown", "id": "c72591f3", "metadata": {}, "source": [ "## Feature provenance\n", "\n", "New mel arrays live in results_v2/features. Original feature arrays in datasets are untouched." ] }, { "cell_type": "code", "execution_count": 5, "id": "062d7a8b", "metadata": { "execution": { "iopub.execute_input": "2026-09-20T16:41:43.357621Z", "iopub.status.busy": "2026-09-20T16:41:43.357502Z", "iopub.status.idle": "2026-09-20T16:41:43.360800Z", "shell.execute_reply": "2026-09-20T16:41:43.360257Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"sample_rate\": 48000,\n", " \"n_fft\": 2048,\n", " \"hop_length\": 1152,\n", " \"n_mels\": 64,\n", " \"fmin\": 0,\n", " \"fmax\": 8000,\n", " \"n_frames\": 256,\n", " \"center\": true,\n", " \"window\": \"hann\",\n", " \"top_db\": 80.0,\n", " \"padding_db\": -80.0,\n", " \"res_type\": \"soxr_hq\"\n", "}\n", "Truncated clips: 0\n", "Native sample rates: {'44100': 360, '48000': 780, '96000': 41, '120000': 202}\n" ] } ], "source": [ "features = json.loads((RESULTS / 'features' / 'config.json').read_text())\n", "print(json.dumps(features['config'], indent=2))\n", "print('Truncated clips:', features['truncated'])\n", "print('Native sample rates:', features['rates'])" ] }, { "cell_type": "markdown", "id": "7855a5dc", "metadata": {}, "source": [ "## Regression checks\n", "\n", "These checks reject overlapping memberships, verify distinct grouped folds, enforce training-only partners, and demonstrate that altering test labels/features cannot affect layer selection." ] }, { "cell_type": "code", "execution_count": 6, "id": "c524094c", "metadata": { "execution": { "iopub.execute_input": "2026-09-20T16:41:43.361901Z", "iopub.status.busy": "2026-09-20T16:41:43.361805Z", "iopub.status.idle": "2026-09-20T16:41:45.276561Z", "shell.execute_reply": "2026-09-20T16:41:45.275953Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "test_ambiguous_and_singleton_id_are_not_targets (__main__.RevisionEvaluationTests.test_ambiguous_and_singleton_id_are_not_targets) ... ok\n", "test_dates_are_not_silently_lost (__main__.RevisionEvaluationTests.test_dates_are_not_silently_lost) ... ok\n", "test_every_task_inherits_disjoint_manifest (__main__.RevisionEvaluationTests.test_every_task_inherits_disjoint_manifest) ... ok\n", "test_grouped_runs_hold_out_distinct_dates (__main__.RevisionEvaluationTests.test_grouped_runs_hold_out_distinct_dates) ... ok\n", "test_layer_selection_cannot_read_test_labels_or_features (__main__.RevisionEvaluationTests.test_layer_selection_cannot_read_test_labels_or_features) ... ok\n", "test_overlap_is_rejected (__main__.RevisionEvaluationTests.test_overlap_is_rejected) ... ok\n", "test_partners_stay_in_training_unit_and_exclude_anchor (__main__.RevisionEvaluationTests.test_partners_stay_in_training_unit_and_exclude_anchor) ... ok\n", "\n", "----------------------------------------------------------------------\n", "Ran 7 tests in 0.212s\n", "\n", "OK\n", "\n" ] } ], "source": [ "command('test_revision_pipeline.py')" ] }, { "cell_type": "markdown", "id": "5a48d41f", "metadata": {}, "source": [ "The registry snapshot is in `registry_v2/pre_revision_materials/`. Run/source hashes and exact training code are under `results_v2/`. See `README_v2.md` for the compatibility aliases retained after moving the completed batch." ] } ], "metadata": { "kernelspec": { "display_name": "Paper revision V2 (.venv)", "language": "python", "name": "review_v2" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.14.7" } }, "nbformat": 4, "nbformat_minor": 5 }