Classifier K-fold leaderboard

Bottom line for the verifier:

The FEV-MCC ensemble wins held-out first_experimental_validation on every metric (FEV MCC 0.319, FEV is_varied MCC 0.598, FEV is_varied exact match 0.199) — beating every single-model checkpoint by 2–4 pp. It picks per-receptor heads from whichever single model maximized FEV MCC for that receptor (R1/R3AF/R3AV from sweep, FcRn6/R2AH/R2AR from v2 baseline, R2B/R3B from retraining_v1).
retraining_v2 sweep is the strongest single model on every FEV is_varied measure and on the in-distribution K-fold is_varied diagnostic (MCC 0.797 ± 0.005). Best choice if a single checkpoint is required.
retraining_v1 wins overall FEV macro MCC among single models (0.298 ± 0.006). Adding costain on top of paper_refiltered+dms gives no net macro-FEV benefit on its own.
paper wins K-fold in-distribution test (MCC 0.961 ± 0.001) but is worst on held-out FEV (0.219). In-distribution test ≠ verifier quality.
The kfold-test-MCC-picked ensemble loses (FEV MCC 0.260) because that criterion picks paper for 4/8 heads, and paper is FEV-weak. The lesson: pick heads on the distribution you want to generalize to, not the one you trained on.

K-fold-trained models

paper

Reproduction of the FcGPT-paper multihead OHE MLP on the paper HF dataset (3.1M rows). Paper hparams (hidden_dim=2048, layers=4, dropout=0.3, lr=5e-5). 5-fold K-fold split (not the paper's LMDB train/val/test).

20260528-212329_classifier_kfold_kfold5_ohe-mlp-v1-paper_seed0

5/5 folds aggregated

retraining_v1

Retraining on paper_refiltered + dms (1.43M rows). Same paper-derived hparams. First merged variant in the retraining cycle.

20260528-212357_classifier_kfold_kfold5_ohe-mlp-v1-retraining-v1_seed0

5/5 folds aggregated

retraining_v2 baseline

Retraining on dms + paper_refiltered + costain (1.47M rows). Same paper-derived hparams. +48,939 costain rows over retraining_v1.

20260528-212408_classifier_kfold_kfold5_ohe-mlp-v1-retraining-v2_seed0

5/5 folds aggregated

retraining_v2 sweep

Retraining on retraining_v2 with Trial #26 hparams from the 200-trial Optuna sweep (dropout=0.1, batch=512; all else identical). Less regularization, larger batches.

20260528-212420_classifier_kfold_kfold5_ohe-mlp-sweep-best-retraining-v2_seed0

5/5 folds aggregated

retraining_v2 shared-multilabel

A single shared multilabel head instead of the 8-trunk multihead, on retraining_v2. Ties the multihead on test MCC (0.946) at ~1/8 the head params, and matches-or-beats the 8-head ensemble on the OOD FEV set; modestly weaker on the in-dist is_varied subgroup (0.753 vs 0.797 MCC).

20260610-153606_classifier_kfold_kfold5_ohe-mlp-shared-multilabel-retraining-v2_seed0

5/5 folds aggregated

Per-receptor ensembles (single-eval, no K-fold variance)

ensemble (kfold-test-mcc picks)

Per-receptor heads chosen by K-fold test MCC. Paper dominates: R1/R3AF/R3AV/R3B from paper, FcRn6/R2AR/R2B from retraining_v1, R2AH from retraining_v2 sweep.

kfold_test_mcc_best

single eval

ensemble (FEV-MCC picks)

Per-receptor heads chosen by single-run first_experimental_validation MCC. R1/R3AF/R3AV from sweep, FcRn6/R2AH/R2AR from v2 baseline, R2B/R3B from retraining_v1.

fev_mcc_best

single eval

Headline test metrics — mean ± std across 5 folds (K-fold models only)

first_experimental_validation (held-out). K-fold models: mean ± std across 5 folds. Ensembles: single-eval (std = 0).

is_varied subgroup on the K-fold test split (sweep wins) — mean ± std across 5 folds

Per-source MCC on K-fold test (retraining variants only — paper is a single source)

Per-receptor MCC on K-fold test — mean ± std across 5 folds

K-fold protocol

	paper	retraining_v1	retraining_v2 baseline	retraining_v2 sweep	retraining_v2 shared-multilabel
MCC	0.9607 ± 0.0005	0.9496 ± 0.0014	0.9347 ± 0.0009	0.9464 ± 0.0012	0.9460 ± 0.0020
Exact match	0.9836 ± 0.0001	0.9791 ± 0.0005	0.9634 ± 0.0004	0.9694 ± 0.0006	0.9694 ± 0.0010
Macro accuracy	0.9890 ± 0.0001	0.9819 ± 0.0006	0.9732 ± 0.0003	0.9782 ± 0.0005	0.9780 ± 0.0008
Macro F1	0.9453 ± 0.0010	0.9424 ± 0.0015	0.9364 ± 0.0012	0.9483 ± 0.0015	0.9474 ± 0.0021

	paper	retraining_v1	retraining_v2 baseline	retraining_v2 sweep	retraining_v2 shared-multilabel	ensemble (kfold-test-mcc picks)	ensemble (FEV-MCC picks)
FEV MCC	0.2192 ± 0.0056	0.2979 ± 0.0064	0.2777 ± 0.0031	0.2848 ± 0.0082	0.3573 ± 0.0131	0.2605 ± 0.0000	0.3193 ± 0.0000
FEV exact match	0.2836 ± 0.0077	0.3682 ± 0.0101	0.3302 ± 0.0057	0.3355 ± 0.0046	0.4195 ± 0.0074	0.3212 ± 0.0000	0.3688 ± 0.0000
FEV macro accuracy	0.5391 ± 0.0155	0.6251 ± 0.0132	0.5748 ± 0.0102	0.5854 ± 0.0084	0.6570 ± 0.0172	0.5856 ± 0.0000	0.6303 ± 0.0000
FEV is_varied MCC	0.5579 ± 0.0237	0.5493 ± 0.0086	0.5502 ± 0.0138	0.5647 ± 0.0220	0.5625 ± 0.0187	0.5457 ± 0.0000	0.5976 ± 0.0000
FEV is_varied exact match	0.1364 ± 0.0179	0.1576 ± 0.0118	0.1709 ± 0.0294	0.1642 ± 0.0236	0.2146 ± 0.0152	0.1325 ± 0.0000	0.1987 ± 0.0000

	paper	retraining_v1	retraining_v2 baseline	retraining_v2 sweep	retraining_v2 shared-multilabel
is_varied MCC	0.7507 ± 0.0095	0.7036 ± 0.0169	0.7788 ± 0.0042	0.7973 ± 0.0052	0.7533 ± 0.0124
is_varied exact match	0.4397 ± 0.0120	0.4665 ± 0.0277	0.6118 ± 0.0050	0.6337 ± 0.0067	0.5782 ± 0.0152
is_varied macro acc	0.8632 ± 0.0055	0.8334 ± 0.0079	0.8721 ± 0.0027	0.8840 ± 0.0028	0.8511 ± 0.0068

	paper	retraining_v1	retraining_v2 baseline	retraining_v2 sweep	retraining_v2 shared-multilabel
dms	—	0.9256 ± 0.0034	0.9052 ± 0.0020	0.9243 ± 0.0016	0.9219 ± 0.0029
paper_refiltered	—	0.9622 ± 0.0011	0.9563 ± 0.0010	0.9641 ± 0.0023	0.9667 ± 0.0022
costain	—	—	0.9215 ± 0.0076	0.9339 ± 0.0080	0.9291 ± 0.0080

	paper	retraining_v1	retraining_v2 baseline	retraining_v2 sweep	retraining_v2 shared-multilabel
R1	0.9781 ± 0.0011	0.9729 ± 0.0025	0.9594 ± 0.0007	0.9661 ± 0.0014	0.9666 ± 0.0017
FcRn6	0.9757 ± 0.0005	0.9786 ± 0.0016	0.9734 ± 0.0026	0.9775 ± 0.0013	0.9775 ± 0.0010
R2AH	0.9051 ± 0.0013	0.8967 ± 0.0077	0.8925 ± 0.0044	0.9140 ± 0.0015	0.9085 ± 0.0023
R2AR	0.9162 ± 0.0036	0.9383 ± 0.0033	0.9209 ± 0.0037	0.9338 ± 0.0037	0.9326 ± 0.0016
R2B	0.9443 ± 0.0032	0.9583 ± 0.0034	0.9343 ± 0.0039	0.9467 ± 0.0051	0.9489 ± 0.0030
R3AF	0.9225 ± 0.0024	0.8975 ± 0.0043	0.8888 ± 0.0087	0.9139 ± 0.0064	0.9138 ± 0.0053
R3AV	0.9367 ± 0.0016	0.9177 ± 0.0040	0.9169 ± 0.0043	0.9357 ± 0.0033	0.9345 ± 0.0034
R3B	0.9319 ± 0.0037	0.8910 ± 0.0061	0.8605 ± 0.0037	0.8809 ± 0.0025	0.8771 ± 0.0089

For each fold k ∈ {0..4}: sha256(seed:seq_hash) mod 5 picks the test fold; the remaining four folds are split 90/10 train/val by an independent hash bucket. Each fold trains from scratch with the same hparams, early-stops on val/loss (patience=10), and reports test metrics at the best validation checkpoint. The wandb group fc_engineering_retraining_kfold hosts all 20 fold runs.