Supervised fine-tuning (Trainfer)
Supervised fine-tuning (Trainfer) is the response-token cross-entropy primitive underlying several project recipes, including Greedy memorization, Chain of Hindsight, and verifier-selected training.
Project status: Registered as sft and weighted_sft. This entry describes the source audit of 14 September 2026; historical measurements retain their original dates.
Mechanism
Chat-template the prompt and response, mask prompt tokens, and minimize response negative log likelihood. The implementation normalizes each sample by its number of unmasked shifted labels and then averages across samples. Weighted SFT computes sum(w_i NLL_i) / sum(w_i), with a small denominator clamp. This changes relative sample importance within a batch.
Implementation and controls
objectives/sft.py contains both functions. Samples contain prompt, response, optional weight, and optional span_prefix. The prefix masks the accepted beginning of a response, leaving the suffix supervised. Returned target positions and token IDs support safety-monitor sidecars.
Rewrite/preferred feedback replay routes to weighted SFT, with a default sample weight of 3.0. A per-sample weight is distinct from the multi-objective coefficient and from learning rate. In a singleton batch a positive weight cancels between numerator and denominator, so increasing it alone does not triple the gradient.
Evidence and evaluation
The historical Track B lattice reports vanilla SFT with 5, 50, and 500 updates at +3.8, +1.2, and +2.5 percentage points, classified as within noise. Other recipes using SFT, including RFT and STaR, damaged their tested baselines. Early five-example memorization results on GSM8K were weaker than matched five-shot ICL. These are different workloads and should not be pooled.
Limitations and interpretation
The term SFT identifies an objective, not a demonstrated continual-learning algorithm. Good training likelihood does not establish free-running recall, paraphrase generalization, or retention. The one-position logit-SGD result in Razin safety is not a guarantee for LoRA/AdamW training. Masking losses changes supervision locations but shared parameters can still affect every output.
Sources
- trainfer: trainfer/objectives/sft.py — checkout audited
1c6391f3773b. - trainfer: trainfer/objectives/_utils.py — checkout audited
1c6391f3773b. - trainfer: trainfer/controller.py — checkout audited
1c6391f3773b. - agi: autoresearch/LATTICE.md — historical revision
3842fd8875ca. - cont: docs/research/JOURNAL.md — checkout audited
87946914c7b9.