Pretraining replay injection
Pretraining replay injection is the proposal to interleave a small amount of general raw-text data with live feedback updates.
Project status: Research proposal/design in the inspected project sources; no implementation or completed local efficacy run identified. This entry describes the source audit of 14 September 2026; historical measurements retain their original dates.
Mechanism
Mix approximately one pretraining-shaped sample per hundred live steps, with the aim of maintaining broad base-model competencies during specialization. This is a replay-data policy, not a new loss; raw text can be trained with NTP.
Implementation and controls
Roadmap PR N proposes a pluggable pretrain_mix.py replay source, a local sample corpus, and an injection ratio of 0.01. Its planned evaluation uses 2,000 feedback events and checks HellaSwag drift, rejecting any task drop above ten percentage points.
Evidence and evaluation
The cited project document records this candidate and its intended experiment. It does not provide a completed local result for this method. Published-paper results mentioned by that document are background, not Trainfer measurements.
Limitations and interpretation
One percent of examples is not necessarily one percent of tokens or compute. A small corpus may poorly represent pretraining and can itself cause drift. The inspected replay scheduler currently reuses feedback records; it does not establish that this pretraining mixture is enabled or implemented.
Sources
- cont: docs/research/production-implementation-roadmap.md — checkout audited
87946914c7b9. - trainfer: trainfer/engine/replay.py — checkout audited
1c6391f3773b.