Phase 5 — three FcGPT-10M base checkpoints (n_embd=512 / n_layer=3), one per training
corpus. All early-stop at epoch 6 with near-identical perplexity (~1.08); the generative behaviour is
what separates them. Green marks the best checkpoint on that row. The all-unknown
-------- prompt is excluded from aggregates — see the note on it below.
runs/20260531-185220_pretrain_fcgpt-10m-v1-paper-reproduction_seed0
runs/20260601-100654_pretrain_fcgpt-10m-v1-retraining-v1_seed0
runs/20260601-134309_pretrain_fcgpt-10m-v1-retraining-v2_seed0
retraining_v2 is the RLXF base. It recovers
paper_reproduction's validity (0.97) and the tightest edit-distance while keeping retraining's lower
memorization, with balanced per-source fit (incl. costain). paper_reproduction conditions slightly
crisper (largest steering gap, below); retraining_v2 wins on the validity / diversity balance. All
three learn valid Fc grammar and genuinely steerable conditioning.
Token-weighted next-token cross-entropy + perplexity on the held-out validation / test splits. Per-source rows show no drift — every source fits in a tight band.
| metric | paper_reproduction | retraining_v1 | retraining_v2 |
|---|---|---|---|
| val loss | 0.0795 | 0.0784 | 0.0777 |
| val perplexity | 1.0827 | 1.0816 | 1.0808 |
| test perplexity | 1.0828 | 1.0816 | 1.0808 |
| per-source val ppl | |||
| costain | — | — | 1.0692 |
| dms | — | 1.0706 | 1.0696 |
| dms+costain | — | — | 1.0485 |
| dms+paper_refiltered | — | 1.0537 | 1.0533 |
| dms+paper_refiltered+costain | — | — | 1.0435 |
| paper_refiltered | — | 1.0849 | 1.0852 |
| paper_refiltered+costain | — | — | 1.0582 |
Per receptor, the verifier's bind-rate on generations conditioned on a profile that
requested binding (1) vs non-binding (0). A positive gap means the
prompt actually steers that receptor. This is the class-balance-immune test: raw "agreement" (next
table) is confounded because per-receptor binder rates are skewed (R3B is a binder in only ~13% of the
data, so a label-blind generator already scores ~0.87 agreement on a 0 request). The gap is
clearly positive on every receptor — conditioning works — but is weakest on R3B (the rarest
binder; the model can barely turn it on even when asked).
| steering gap · bind|req=1 − bind|req=0 | paper_reproduction | retraining_v1 | retraining_v2 |
|---|---|---|---|
| mean gap (all receptors) | 0.44 | 0.37 | 0.39 |
| R1 | 0.77 | 0.67 | 0.58 |
| FcRn6 | 0.60 | 0.52 | 0.59 |
| R2AH | 0.46 | 0.32 | 0.27 |
| R2AR | 0.52 | 0.34 | 0.37 |
| R2B | 0.41 | 0.36 | 0.38 |
| R3AF | 0.29 | 0.30 | 0.44 |
| R3AV | 0.33 | 0.32 | 0.31 |
| R3B | 0.14 | 0.14 | 0.14 |
Sampled across the 58-prompt fixed panel (excl. base_lm). Agreement =
per-receptor match between the verifier ensemble's calls and the requested profile (note: not a
steering measure on its own — the majority-class baseline averages ~0.73; read the steering gap above
instead); full-8 match = fraction matching all specified receptors; memorization
= generations exactly equal to a training sequence (by seq_hash).
| panel aggregate | paper_reproduction | retraining_v1 | retraining_v2 |
|---|---|---|---|
| agreement | 0.77 | 0.69 | 0.72 |
| full-8 match | 0.33 | 0.25 | 0.27 |
| valid rate | 97% | 88% | 97% |
| memorization | 0.64 | 0.52 | 0.53 |
| duplicate | 0.36 | 0.36 | 0.44 |
| mean edit-to-WT | 3.3 | 1.9 | 1.3 |
Per-receptor classifier agreement broken out by prompt intent. High values on "mostly-1" groups partly reflect the WT-like base distribution binding most receptors, not steering per se — the steering-gap table is the unconfounded read.
| per-receptor agreement · by prompt group | paper_reproduction | retraining_v1 | retraining_v2 |
|---|---|---|---|
| single_positive | 0.82 | 0.77 | 0.76 |
| single_negative | 0.77 | 0.79 | 0.79 |
| paper | 0.89 | 0.60 | 0.66 |
| fev_design | 0.70 | 0.66 | 0.67 |
| dms_informed | 0.89 | 0.74 | 0.86 |
| rare | 0.63 | 0.60 | 0.61 |
| base_lm | — | — | — |
-------- promptIt is not out-of-distribution: the training corpora do contain all-pad rows (~0.12% in retraining_v2) and ~67% of rows specify only one receptor, so partial/missing prompts are the norm. But with zero yes/no tokens the model loses its length and WT-structure prior — it emits valid amino acids (~94%) that are the wrong length (15–146, never 232) and not WT-anchored, i.e. structurally incoherent Fc fragments rather than full-length sequences. So the all-unknown condition (severely under-represented at 0.12%) is a genuine weak spot for sampling the base distribution, not an artifact — worth knowing since unconditional sampling is the natural pre-RLXF baseline.